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25th  Test  and  Evaluation  National  Conference 

“New  Administration,  New  Opportunities  ” 

Atlantic  City,  NJ 
2-5  March  2009 


Agenda 


Monday,  2  March  2009 


WORKSHOP  I 

MISSION-BASED  TEST  &  EVALUATION  (MBTE) 
Presenters:  Mr.  Chris  Wilcox,  AEC 

•  Mr.  Jack  Sheehan,  ORSA  Corporation 
PM  Presenters: 

•  Mr.  Jack  Sheehan,  ORSA  Corporation 


Tuesday,  3  March  2009 

CONFERENCE  KEYNOTE  ADDRESS 

Mr.  Keith  Williams,  CEO,  Underwriters  Laboratories  (UL) 

DOT&E  LOOKING  AHEAD 

Honorable  Charles  McQueary,  Director,  Operational  Test  &  Evaluation  (DOT&E) 

ROUNDTABLE  :  REALITIES  FACING  THE  WORLD  OF  DEFENSE  AND  HOMELAND  SECURITY  T&E 
Moderator:  Maj  Gen  Steve  Sargeant,  USAF,  Commander  AFOTEC 

GAO  ASSESSSMENT  OF  DOD’S  MAJOR  WEAPON  SYSTEMS  ACQUISITION  PORTFOLIO 

Mr.  Ronald  Schwenn,  US  Government  Accountability  Office 


SESSION  A:  MISSION  CAPABILITY  TESTING 

•  “ Evaluating  the  Mission  -  Translating  Systems  Performance  to  Unit  Capabilities  ”,  Mr.  David  Thomas,  Round  Table  Defense,  LLC. 

•  “System  of  Systems  -  Survivability,  Lethality,  Vulnerability  Assessment”,  Mr.  Jeffrey  A.  Smith  and  Ms.  Beth  Ward,  U.S.  Army  Research  Lab 

•  “Structuring  T&E  for  Validation  of  Complex  Systems  Capabilities  and  Exploration  of  Emergent  Behaviors  ”,  Mr.  Joseph  Tribble,  AVW 
Technologies 

•  “Using  IMPRINT  to  Translate  Human  Performance  Into  Mission  Effectiveness  to  Focus  Test  and  Evaluation  ”,  Ms.  Diane  Mitchell  &  Chameta 
Samms 

SESSION  B-NEW  T&E  POLICY  AND  POLICY  IMPLEMENTATION 

•  “New  Test  and  Evaluation  Master  Plan  Guidance”,  Ms.  Darlene  Mosser-Kemer,  OUSD  (AT&L) 

•  “Walking  the  Line  With  Title  10:  Implementation  Strategies  for  Integrated  Testing”,  Dr.  Beth  Wilson,  Raytheon 

•  “Systems  Assurance  Test  and  Evaluation  in  Defense  Acquisition  -  Specification,  Planning  and  Resourcing” ,  Ms.  Christine  Hines,  OUSD 
(AT&L) 

•  “Automated  GPS  Simulation  Test  Process  ”,  2nd  LT  Matthew  Steele,  Holloman  AFB 


2009test.html[4/22/2016  2:50:22  PM] 


Untitled  Document 


SESSION  C-T&E  METHODOLOGY  AND  M&S 

•  “ How  to  Incorporate  Modeling  and  Simulation  into  the  System  Acquisition  Life  Cycle  ”,  Mr.  Derek  Kropp,  Navy  Patuxent  River 

•  “Extending  Test  and  Evaluation  Modeling  and  Simulation  Capabilities  with  Gaming  Technology”,  Mr.  Geoffrey  Robinson,  Ft.  Hood 

•  “Integrating  the  System  Engineering  “V”  in  a  System  of  Systems  ”,  Mr.  Jack  Sheehan,  ORSA  Corporation 

PRESENTATION  OF  THE  2009  WALTER  W.  HOLLIS  AWARD  FOR  OUTSTANDING  LIFETIME  ACHIEVEMENT  IN  DEFENSE  TEST 
&  EVALUATION. 

Guest  Speaker:  Mr.  Winslow  Wheeler,  Director  of  the  Straus  Military  Reform  Project  of  the  Center  for  Defense  Information 

Wednesday,,  4  March  2009 


SESSION  IV:  T&E  POLICY  &  STUDIES 

•  “Physics  of  Failure  -  the  Critical  Path  to  Saving  $M  in  T&E”,  Dr.  W.  Forrest  Crain,  Director,  US  AMSAA 

•  “A  Common  Platform  for  DOD  and  DHS  to  Partner  in  the  T&E  Community”,  Dr.  Richard  Murrow,  CEO,  Air  Academy  Associates 

SESSION  VI:  T&E  WORKFORCE  &  RESOURCES 
Session  VI  Chair:  Mr.  Charles  Larson,  SURVICE  Engineering 

•  “ The  DoD  Strategic  Plan  for  Test  &  Evaluation  Resources ”,  Mr.  Jason  Coker,  OUSD  (AT&L/TRMC) 

•  “Today ’s  and  Tomorrow ’s  DoD  Test  &  Evaluation  Acquisition  Workforce ”,  Ms.  Darlene  Mosser-Kerner,  OUSD  (AT&L) 

SESSION  D-RESOURCES,  INSTRUMENTATION,  AND  TARGETS 
Session  VI  Chair:  Mr.  Charles  Larson,  SURVICE  Engineering 

•  “Test  and  Training  Enabling  Architecture,  TENA,  An  Important  Component  in  Joint  Mission  Environment  Test  Capability  Successes  ”,  Mr. 
Gene  Hudgins,  BAE  Systems 

•  “Joint Mission  Environment  Test  Capability” ,  Mr.  Chip  Ferguson,  TRMC 

•  “Evolving  Threats  -  How  They  Impact  T&E  Testing  and  Infrastructure  ”,  Mr.  Dick  Dickson,  TYBRIN  Corporation 

•  “Threat/T arget  Simulation  for  Live-Fire  Events  ”,  Mr.  Greg  Van  Splinter,  NAWCWD 

SESSION  E-  APPLICATION  OF  T&E/POLICY  TO  SPECIFIC  SYSTEMS 

•  “Challenges  Facing  T&E  of  Systems  to  Combat  Nuclear  and  Radiological  Smuggling” ,  Mr.  Stephen  DeFrank,  Jr.,  Mantech  SRS,  Inc. 

•  “C-RAM Delivers  on  Promises  and  Saves  Lives  ”,  Mr.  James  Bloodsworth,  Army  C-RAM  Program  Directorate 

Thursday,,  5  March  2009 

MISSION-BASED  TEST  &  EVALUATION  STRATEGY: 

•  “An  Inter-Agency-Developed  Process  to  Link  Mission  Capability  With  System  Functional  Requirements”,  Mr.  Christopher  Wilcox,  ATEC 

•  “Evaluating  An  Immersive  Virtual  World  For  Accelerating  T&E”,  Mr.  Redge  Bartholomew,  Rockwell  Collins 

FCS  CAPABILITIES  TO  EMPOWER  THE  SOLDIER— WHAT  IS  AND  WHAT  WILL  BE 

Mr.  Bud  Irish,  Vice  President,  SAIC 

THE  ANALYSIS  OF  ALTERNATIVES  (AO A):  A  MISSION-ORIENTED,  EVALUATION-BASED  FRAMEWORK  FOR  DEFENSE  TEST 
&  EVALUATION 
Mr.  Vincent  P.  Roske,  Jr.,  IDA 

CONFERENCE  SYNOPSIS:  OUTBRIEFS  OF  BREAKOUT  /  WRITING  SESSIONS 

Improving  T&E  Efficiency  and  M&S  -  Dr.  Paul  Deitz,  ARL  and  Dr.  Mark  Kiemele,  Air  Academy  Associates 


2009test.html[4/22/2016  2:50:22  PM] 


National  Defense  Industrial  Association 


'  T* 


. 


PROMOTING  NATIONAL  SECURITY  SINCE  1919 

25™  ANNUAL 

TEST  & 


arzam 


NATIONAL  CONFERENCE: 

“New  Administration, 
New  Opportunities” 


WHAT  YOU  DON’T  WANT  TO 
MISS  AT  THIS  CONFERENCE: 


Internationally  known  civilian 
and  military  authorities 
presenting  their  views  on 
Defense  and  Homeland  Security 
T&E  issues 


Outstanding  Tutorials 

Presentation  of  the  2009  Hollis 
Award  and  Tester  of  the  Year 
Awards 

Viewpoints  from  the  Media/ 
TV/  Authors  Informative 
Displays  from  Vendors  in  T&E- 
related  fields 

Multiple  Breakout  Sessions  to 
Address  Specific  Pacing  Issues 

Panel  Discussions  Assessing 
New  Policy  and  Issues 

Over  70  presenters  addressing 
Test  Facilities,  Congressional 
Actions,  Study  Results,  Specific 
T&E  Applications  to  Defense 
and  Homeland  Security 


MARCH  2-5, 2009 
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SHERATON  ATLANTIC  CITY  ►  ATLANTIC  CITY,  NJ 


EVENT  #9910 


TEST  &  EVALUATION 
GENERAL  INFORMATION 


CONFERENCE  ANNOUNCEMENT 

This  25th  Annual  National  Test  &  Evaluation  Conference  is  sponsored  by  the  NDIA 
T&E  Division  and  supported  by  the  Office  of  the  Under  Secretary  of  Defense  (AT&T) 
and  the  Director,  Operational  Test  &  Evaluation  (DOT&E)  in  cooperation  with  ASTM. 

The  conference  will  feature  nationally  known  leaders  in  the  field  of  National  Defense  and 
Homeland  Security,  military  leaders  from  the  combat  zones,  senior  industry  leaders,  authorities 
on  the  conference  topics  from  academia  and  policy  study  houses,  Congressional  leaders,  as 
well  as  a  host  of  individuals  charged  with  conducting  test  and  evaluation  on  an  ongoing  basis. 

The  timing  of  this  conference  also  permits  meaningful  discussion  on  recent  legislative  actions 
and  Defense  Acquisition  policy  initiatives  as  well  as  the  results,  implications  and  implementation 
of  such  recent  studies  as  the  Defense  Science  Board  Study  on  Test  &  Evaluation,  DAPA  and 
recommendations.  While  a  new  Administration  offers  a  unique  window  of  opportunity  to  invite  new 
ideas  and  changes  in  approaches  to  T&E,  it  is  also  vital  that  approaches  that  have  been  demonstrated 
to  be  responsive  and  effective  not  be  sacrificed  on  the  altar  of  novelty  or  simply  “change  for  the 
sake  of  change.”  This  conference  will  feature  open  discussion  and  debate  addressing  what’s  working 
and  should  be  retained  and  what’s  not  and  suggestions  for  change  to  correct  these  deficiencies. 

The  full  scope  of  issues  related  to  and  impacted  by  defense  and  homeland  security  test  and 
evaluation  will  be  addressed  at  this  event,  with  emphasis  on  emerging  technologies.  Topics  will 
include  acquisition  and  T&E  policy  and  procedures,  related  studies,  T&E  infrastructure  issues, 
implications  of  BRAC  on  the  T&E  community,  modeling  and  simulation  in  support  of  T&E, 
training  and  upgrading  the  T&E  workforce,  T&E  in  support  of  rapid  deployment,  pressing 
T&E  environmental  issues,  novel  T&E  concepts,  T&E  encroachment  issues,  implications  of 
feedback  from  the  combat  zones,  test  design  approaches,  and  a  host  of  other  related  topics. 

Representatives  of  the  U.S.  and  international  communities,  including  government,  industry,  academia 
and  other  nonprofit  institutions  are  invited  to  participate. 


CONFERENCE  ATTIRE 

Conference  attire  is  business  for  civilians  and  Class  A  uniform  for  military.  In  addition,  your 
identification  badge,  received  upon  conference  check-in,  must  be  worn  at  all  times. 


NDIA  T&E  EXECUTIVE  BOARD: 

Mr.  Joe  Andrese,  APG  NDIA  Chapter 
Mr.  Dennis  Bely,  ARL 
Dr.  Keith  Bradley,  LLNL 
Mr.  Britt  Bray,  DRC  Corporation 
Mr.  Sam  Campagna,  NDIA 

RADM  David  Crocker,  USN  (Ret),  Hard  Charger  Solutions 

Dr.  Paul  Deitz,  HRED ,  ARL 

Mr.  Dick  Dickson,  Tybrin  Corporation 

Mr.  Russ  Hauck,  Simulation  Information  Systems 

Dr.  Anne  Hillegas,  ARA  Corporation 

Mr.  John  Illgen,  Northrop  Grumman  Corporation 


RADM  Charles  “Bert”  Johnston,  USN  (Ret),  Wyle  Labs 

Dr.  Mark  Kiemele,  Air  Academy  Associates 

Mr.  Chuck  Larson,  SURVICE  Engineering 

Mr.  James  O’Bryon,  The  OBryon  Group ,  T&E  Division  Chair 

Mr.  Brendan  Rhatigan,  Lockheed  Martin  Corporation 

Dr.  Ernest  Seglie,  DOT&E,  OSD 

Mr.  Jack  Sheehan,  ARL 

Dr.  Lowell  Tonnessen,  IDA 

Dr.  Juan  Vitali,  JPEOCBD 

Mr.  William  Yeakel,  ORSA  Corporation 


TEST  &  EVALUATION 
AWARD  INFORMATION 


AWARD  INFORMATION 

WALTER  W.  HOLLIS  HONORS  BANQUET 

The  Walter  W.  Hollis  Award  is  presented  annually  in  recognition  of  lifetime  contributions  and  achievement 
in  the  area  of  defense  Test  &  Evaluation.  The  award  is  presented  in  the  name  of  Walter  W.  Hollis  who 
is  recognized  for  his  dedicated  and  long-standing  service  and  contributions  in  the  field  of  Test  & 
Evaluation. 

Previous  Recipients  of  this  award: 

Dr.  Paul  H.  Deitz,  Acting  Director  HRED,  ARL,  APG,  MD  (2008) 

Mr.  James  F.  OBryon ,  Former  DOT&E  /  LFT  (2007) 

RADM  Charles  “Bert” Johnston,  USN  (Ret),  Wyle  Laboratories  (2006) 

Hon  Thomas  Christie,  DOT&E,  OSD  (2005) 

Dr.  Marion  Williams,  HQAFOTEC  (2004) 

Mr.  James  Fasig,  Aberdeen  Test  Center  (2003) 

Mr.  G.  Thomas  Castino,  Underwriters  Laboratories,  Inc.  (2002) 

Hon  Philip  Coyle,  III,  DOT&E,  OSD  (2001) 

Mr.  Walter  Hollis,  Department  of  the  Army  (2000) 


TESTER  OF  THE  YEAR  AWARDS 

These  awards,  presented  to  outstanding  individuals  in  the  field  of  Test  &  Evaluation,  offer  OSD  and  each 
Military  Service  Test  &  Evaluation  Department  the  opportunity  to  select  three  award  recipients  for  recognition 
as  the  Tester  of  the  Year  in  specific  categories.  The  three  categories  recognized  are:  Military,  Civilian,  and 
Contractor.  Recipients  will  be  recognized  at  the  conference  Awards  Luncheon  on  Wednesday,  March  4. 


2008  TESTER  OF  THE  YEAR  AWARDEES 


MAJ  Theotis  Clemons,  USA 

Army  Military  Tester  of  the  Year,  2008 


Mr.  Blair  L.  Bozek 

USAF  Contractor  Tester  of  the  Year,  2008 


Mr.  W.  Scott  Walton 

Army  Civilian  Tester  of  the  Year,  2008 

Mr.  Timothy  J.  Conway 

Army  Contractor  Tester  of  the  Year,  2008 

Col  Eileen  A.  Bjorkman,  USAF 
OSD  Military  Tester  of  the  Year,  2008 

Mr.  William  J.  Kralik 

OSD  Civilian  Tester  of  the  Year,  2008 

Mr.  Thomas  G.  Kane 

OSD  Contractor  Tester  of  the  Year,  2008 


LT Leonard L.  Adams,  Jr.,  USN 
Navy  Military  Tester  of  the  Year,  2008 

Mr.  Kevin  M.  Ransford 

Navy  Civilian  Tester  of  the  Year,  2008 

Dr.  Jay  R.  Smith 

Navy  Contractor  Tester  of  the  Year,  2008 

Maj  Garrett  L.  Benson,  USMC 

Marine  Corps  Military  Tester  of  the  Year,  2008 

Mr.  John  Lee 

Marine  Corps  Civilian  Tester  of  the  Year,  2008 


Capt  James  D.  Conley,  USAF 
USAF  Military  Tester  of  the  Year,  2008 

Mr.  Kenneth  E.  Bandy 

USAF  Civilian  Tester  of  the  Year,  2008 


Mr.  Donald  C.  Mueller 

Marine  Corps  Contractor  Tester  of  the  Year,  2008 

Joint  Electronic  Protection  Air  Combat  Joint  T&E  Team 
OSD  Special  Category  Tester  of  the  Year,  2008 


TEST  &  EVALUATION 
AGENDA:  MARCH  02, 2009 


MONDAY,  MARCH  2,  2009 


11:00  AM  -  5:00  PM  CONFERENCE  REGISTRATION 

TUTORIAL  SESSIONS 


WORKSHOP  1 

WORKSHOP  ll-(l 

1:30  PM 

MISSION-BASED  TEST  &  EVALUATION  (MBTE) 

Presenters:  Mr.  Chris  Wilcox,  AEC 

Mr.  Britt  Bray,  DRC  Corporation 

Mr.  Jack  Sheehan,  ORSA  Corporation 

THE  SECRET  TO  SURVIVING  &  THRIVING  IN  A 

DOWN-TURN  ECONOMY 

Presenter:  Dr.  Mark  J.  Kiemele,  President  and  Co-founder,  Air 
Academy  Associates 

3:00  PM 

AFTERNOON  BREAK 

AFTERNOON  BREAK 

3:15  PM 

MISSION-BASED  TEST  &  EVALUATION  (MBTE) 
CONTINUED 

Presenters:  Mr.  Chris  Wilcox,  AEC 

Mr.  Britt  Bray,  DRC  Corporation 

Mr.  Jack  Sheehan,  ORSA  Corporation 

THE  SECRET  TO  SURVIVING  &  THRIVING  IN  A 

DOWN-TURN  ECONOMY  CONTINUED 

Presenter:  Dr.  Mark  J.  Kiemele,  President  and  Co-founder,  Air 
Academy  Associates 

TUTORIAL  FEE  $50 

TUTORIAL  DESCRIPTIONS 

WORKSHOP  I 

Moderator:  Dr.  Paul  Deitz,  Acting  Director,  HRED  ARL 

MISSION-BASED  TEST  &  EVALUATION  (MBTE) 

Topics  to  be  covered  include  a  synopsis  of  MBTE  Efforts, 
Accomplishments  of  the  AEC  MB  T&E  Program  and  Logic 
and  Approach  for  Bottom-Up  MBTE  Execution  Strategy. 

We  now  note  that  OSD  P&R  has  developed  a  process  utilizing 
MB  Tasks/  Conditions/  Standards  (T/C/S)  “criteria”  or  a  kind 
of  “use  case  construct”.  Sets  of  T/C/S  are  used  as  surrogates  for 
fully  developed  missions  (or  even  vignettes),  but  are  directly 
relatable  to  the  materiel  readiness  and  human  skill  sets  most 
relevant  to  the  mission(s)  contemplated.  Furthermore,  OSDs 
P&R  Office  has  a  roll  up  process  by  means  of  which  the  statuses 
of  both  materiel  and  people  capabilities  can  be  related  to 
particular  mission  sets.  Our  aim  is  to  collect  the  details  of  these 
current  practices,  and  then  emulate/apply  them  as  possible 
to  both  MB  T&E  and  MB  Technology  Assessment  (TA). 

MISSION-BASED  TEST  &  EVALUATION  (MBTE)  CONTINUED 

Conclusion  to  MBTE  Presentation  as  well  as  a  Working 
&  Writing  Session  to  permit  time  for  Q&A,  discussion  and 
drafting  of  a  paper  addressing  issues  arising  from  this  tutorial 
including  recommendations  for  implementation  of  the  MBTE 
concept. 


WORKSHOP  II 

Moderator:  Mr.  William  Yeakel,  ORSA  Corporation 

THE  SECRET  TO  SURVIVING  &  THRIVING  IN  A  DOWN-TURN  ECONOMY 

A  leading  question  in  todays  marketplace  is  what’s  the  best 
investment  a  company  can  make?  This  workshop  will  introduce 
the  participant  to  a  methodology  that  can  be  used  to  systematically 
generate  breakthrough  improvement  and  deliver  double-digit  return 
on  investment.  If  your  Six  Sigma  or  Lean  Six  Sigma  (LSS)  initiative 
has  fallen  short  of  the  value  you  were  expecting  and  needs  a  shot  in 
the  arm  to  rejuvenate  it,  or  if  you  are  a  newcomer  to  LSS  or  Design 
for  Six  Sigma  (DFSS),  this  workshop  is  for  you.  This  session  will 
provide  you  with  an  overview  of  the  necessary  methodologies, 
infrastructure,  and  approach  that  have  proven  to  be  valuable 
breakthrough  performance  generators  for  a  variety  of  companies. 
Companies  no  longer  can  afford  to  depend  on  the  epiphanies  of  a 
select  creative  few  to  achieve  innovative  breakthroughs.  Although 
there  are  no  magic  formulas  or  silver  bullets,  this  session  will  give 
you  the  basics  of  reliable,  predictable,  and  repeatable  breakthroughs 
by  providing  you  a  systematic  approach  to  gaining  order-of- 
magnitude  return  on  investment. 

THE  SECRET  TO  SURVIVING  &  THRIVING  IN  A  DOWN-TURN  ECONOMY 
CONTINUED 

Conclusion  to  discussions  of  LSS  and  DFSS  and  their  applications 
as  well  as  a  Working  &  Writing  Session  to  permit  time  for  Q&A, 
discussion  and  drafting  of  a  paper  addressing  issues  arising  from 
this  tutorial  including  recommendations  for  implementation  of 
these  applications  to  DoD’s  and  DHS’s  missions. 


5:00  PM  -  6:00  PM  EVENING  RECEPTION  IN  DISPLAY  AREA-(PEARL  BALLROOM) 
6:00  PM  CONFERENCE  ADJOURNED  FOR  THE  DAY  &  DISPLAY  AREA  CLOSED 


TEST  &  EVALUATION 
AGENDA:  MARCH  03,  2009 


TUESDAY,  MARCH  3,  2009 

7:00  AM  -  5:00  PM  CONFERENCE  REGISTRATION 

7:00  AM  -  8:00  AM  CONTINENTAL  BREAKFAST  IN  DISPLAY  AREA 

8:00  AM  CALL  TO  ORDER  AND  REMARKS-(CR0WN  BALLROOM  3, 4  &  5) 

►  Mr.  Sam  Campagna,  Director,  Operations,  NDIA 

8:05  AM  TRIBUTE  TO  OUR  NATION  AND  WARFIGHTERS,  NATIONAL  ANTHEM 

8:10  AM  WELCOME  AND  CONFERENCE  INTRODUCTORY  REMARKS 

►  Mr.  James  O’Bryon,  Chairman,  NDIA  T&E Division;The  O’Bryon  Group 

SESSION  I 

Session  I  Chair:  Mr.  James  O’Bryon,  Chairman,  NDIA  T&E  Division 

8:15  AM  CONFERENCE  KEYNOTE  ADDRESS 

►  Mr.  Keith  Williams,  CEO,  Underwriters  Laboratories  (UL) 

The  UL  seal  is  one  of  the  most  recognizable  trademarks  in  the  world.  Since  1894,  the  UL  has  been  at  the  forefront  of  testing 
and  testing  policy  to  assure  that  systems  not  only  work  effectively  but  also  that  they  will  be  reliable  and  safe  to  operate  over 
the  long  haul.  It’s  an  honor  to  have  this  leader  of  the  world’s  most  prominent  and  respected  T&E  organization  with  us  at  this 
Silver  Anniversary  ofNDIA’s  T&E  Division. 

9:00  AM  D0T&E  LOOKING  AHEAD 

►  Honorable  Charles  McQueary,  Director,  Operational  Test  &  Evaluation  (DOT&E) 

9:30  AM  PERSPECTIVES  FROM  A  FORMER  TEST  PILOT  AND  FORMER  D,  0T&E 

►  Honorable  John  “Jack”  Krings,  President,  Krings  Corporation,  Former  DOT&E 

10:00  AM  MORNING  BREAK  &  NETWORKING  IN  DISPLAY  AREA 
SESSION  II 

Session  II  Chair:  RADM  Charles  “Bert”  Johnston,  USN  (Ret),  Wyle  Labs 

10:30  AM  ROUNDTABLE:  REALITIES  FACING  THE  WORLD  OF  DEFENSE  AND  HOMELAND  SECURITY  T&E 

Moderator:  Maj  Gen  Steve  Sargeant,  USAF,  Commander  AFOTEC 

MG  Roger  Nadeau,  USA,  Commander  ATEC 

RDML  David  “Decoy”  Dunaway,  USN,  Commander,  OPTEVFOR 

Col  David  Reeves,  USMC,  Director,  MCOTEA 

Mr.  George  Ryan,  Director,  OT&E,  DHS 

11:45  AM  -  1:15  PM  LUNCHEON  AND  SPECIAL  GUEST  SPEAKER-(CR0WN  BALLROOM  1  &  2) 

►  Mr.  Vago  Muradian,  Editor,  Defense  News  and  TV  Host  of  CBS’s  “ This  Week  in  Defense  News” 


TEST  &  EVALUATION 
AGENDA:  MARCH  03,  2009 


SESSION  III 

Session  III  Chair:  Mr.  Jack  Sheehan,  ORSA  Corporation 

1:30  PM  GAO  ASSESSMENT  OF  DOD’S  MAJOR  WEAPON  SYSTEMS  ACQUISITION  PORTFOLIO 

►  Mr.  Ronald  Schwenn,  US  Government  Accountability  Office 

2:00  PM  NDIAIC0TE  ANNUAL  REPORT 

►  Mr.  Larry  Graviss,  ICOTE  Chain  President ,  Eagle  Engineering 

2:30  PM  AFTERNOON  BREAK  &  NETWORKING  IN  DISPLAY  AREA 

3:00  PM  -  5:30  PM  CONCURRENT  FOCUS  AND  WORKING  DRAFT  SESSIONS 

Each  Breakout  Session  will  consist  of  both  oral  presentations  and  collaboration  between  presenters  and  audience  to  draft  the 
recommendations  and  issues  raised  during  each  session  for  presentation  to  the  plenary  session  Thursday  morning.  The  Session 
host  and  presenters  will  be  responsible  for  pulling  these  issues  together  for  later  presentation. 


SESSION  A- (STEEL  PIER) 

SESSION  B-(CR0WN  BALLROOM  4) 

SESSION  C- (CROWN  BALLROOM  5) 

MISSION  CAPABILITY  TESTING 
SESSION 

NEW  T&E  POLICY  AND  POLICY 
IMPLEMENTATION 

T&E  METHODOLOGY 

AND  M&S 

Chair:  Mr.  Britt  Bray, 

DRC  Corporation 

Chair:  Mr.  Robert  Wojciechowski, 
APG 

Chair:  Dr.  Mark  Kiemele, 

Air  Academy  Associates 

3:00  PM 

Evaluating  the  Mission  -  Translating 
Systems  Performance  to  Unit 
Capabilities 

Mr.  David  Thomas,  RoundTable  Dtfense,  LLC. 

New  Test  and  Evaluation  Master  Plan 
Guidance 

Ms.  Darlene  Mosser-Kerner,  OUSD 
(AT&L) 

Early  Decision  Analysis  Guiding 

System  Reliability  Growth 

Dr.  Patricia  Jacobs,  Naval  Postgraduate 
School 

3:30  PM 

System  of  Systems  -  Survivability, 
Lethality,  Vulnerability  Assessment 

ITAA  GEAI-STD-0009  —  New  Best 
Practices  Standard  for  Reliability 
Assessment  and  Verification 

How  to  Incorporate  Modeling 
and  Simulation  into  the  System 
Acquisition  Life  Cycle 

Ms.  Beth  Ward,  U.S.  Army  Research  Lab 

Mr.  Enzo  Long,  DOT&E '  OSD 

Dr.  Derek  Kropp,  Navy  Patuxent  River 

4:00  PM 

Structuring  T &E  for  Validation  of 
Complex  Systems  Capabilities  and 
Exploration  of  Emergent  Behaviors 

Walking  the  Line  With  Title  10: 
Implementation  Strategies  for  Integrated 
Testing 

Extending  Test  and  Evaluation 
Modeling  and  Simulation  Capabilities 
with  Gaming  Technology 

Mr.  Joseph  Tribble,  AVW  Technologies 

Dr.  Beth  Wilson,  Raytheon 

Mr.  Geoffrey  Robinson,  Ft.  Hood 

4:30  PM 

Using  IMPRINT  to  Translate 

Human  Performance  Into  Mission 
Effectiveness  to  Focus  Test  and 
Evaluation 

Systems  Assurance  Test  and  Evaluation 
in  Defense  Acquisition  -  Specification, 
Planning  and  Resourcing 

Integrating  the  System  Engineering 
“V”  in  a  Sytem  of  Systems 

Ms. Diane  Mitchell  &  Charneta  Samms 

Ms.  Christine  Hines,  OUSD  (AT&L) 

Mr.  Jack  Sheehan,  ORSA  Corporation 

5:00  PM 

Automated  GPS  Simulation  Test 

Process 

Applying  the  SCORE  Framework 
to  Evaluate  Advanced  Military 
Technologies 

2nd  LT  Matthew  Steele,  Holloman  AFB 

Mr.  Craig  Schlenoff,  NLST 

5:30  PM  AFTERNOON  SESSION  COMPLETE 


TEST  &  EVALUATION 
AGENDA:  MARCH  03,  2009 


6:30  PM  -  8:30PM  WALTER  W.  HOLLIS  HONORS  BANQUET-(CROWN  BALLROOM  1  &  2) 

PRESENTATION  OF  THE  2009  WALTER  W.  HOLLIS  AWARD  FOR  OUTSTANDING  LIFETIME  ACHIEVEMENT  IN  DEFENSE  TEST  & 
EVALUATION. 

The  Walter  W.  Hollis  Award  is  presented  annually  in  recognition  of  lifetime  contributions  and  achievement  in  the  area  of  defense 
Test  &  Evaluation.  The  award  is  presented  in  the  name  of  Walter  W.  Hollis  who  is  recognized 
for  his  dedicated  and  long-standing  service  in  the  field  of  Test  &  Evaluation. 

Recipient:  Dr.  Ernest  Seglie,  Science  Advisor  to  the  Director,  Operational  Test  &  Evaluation,  OSD 

Dr.  Seglie  received  his  B.S.  at  The  Cooper  Union  and  his  Ph.D.  in  Physics  from  the  University  of 
Massachusetts.  After  teaching  and  research  assignments  at  Rensselaer  Polytechnic  Institute  and 
Yale  University  he  joined  the  Institute  for  Defense  Analyses.  He  led  the  IDA  analysis  for  the  Joint 
Test  TASVAL,  Tactical  Aircraft  Survivability  Evaluation,  which  was  the  largest  field  test  conducted 
up  to  that  time.  He  was  project  leader  for  IDAs  support  of  the  Sgt.  York  Operational  Test,  the 
first  test  overseen  by  the  then  just  established  Office  of  Director,  Operational  Test  and  Evaluation 
(DOT&E).  He  received  the  Andrew  Goodpaster  Award  for  Excellence  in  Research  from  IDA.  He 
became  the  first  Science  Advisor  to  DOT&E  in  1988  and  his  efforts  have  been  rewarded  with  the 
Civilian  Service  Medal,  the  Secretary  of  Defense  Medal  for  Meritorious  Service  (twice)  and  the 
rank  of  Meritorious  Senior  Professional,  conferred  on  him  in  2003  by  President  Bush. 


GUEST  SPEAKER 

Mr.  Winslow  Wheeler,  Director  of  the  Straus  Military  Reform  Project  of  the  Center  for  Defense  Information 

“Mr.  Winslow  Wheeler  is  widely  known  in  defense  circles  for  working  on  national  security  issues  for  members  of  the  US  Senate 
and  the  GAO.  In  the  Senate,  Wheeler  advised  Senators  Jacob  K.  Javits  (R-NY),  Nancy  L.  Kassebaum  (R-KS),  David  Pryor  (D-AK) 
and  Pete  V.  Domenici  (R-NM).  He  was  the  first  and,  according  to  Senate  records  the  last,  Senate  staffer  to  work  simultaneously 
on  the  personal  staffs  of  a  Republican  and  a  Democrat.  He  has  also  appeared  as  a  guest  on  defense  matters  for  C-SPAN,  NPR, 

PBS,  60  Minutes,  Armed  Forces  Journal,  Barron’s,  the  Washington  Post,  the  Politico  and  a  number  of  other  media  outlets. 

He  has  also  published  on  a  wide  range  of  defense  topics.  In  2002,  while  working  on  the  Republican  staff  of  the  Senate  Budget  Committee, 
he  authored  an  essay  under  the  pseudonym  “Spartacus,”  addressing  Congress’  reaction  to  the  September  11, 2001  terrorist  attacks.  He 
has  authored  two  books:  “The  Wastrels  of  Defense”  and  “Military  Reform”  and  released  a  new  anthology,  “America’s  Defense  Meltdown”. 

*The  cost  of  this  Honors  Banquet  is  included  in  the  registration  fee.  Guests  of  conference  registrants  are  welcome  for  an  additional  fee 
of  $75. 


Dr.  Ernest  Seglie 
Science  Advisor  to  the  Director, 
Operational  Test  &  Evaluation,  OSD 


TEST  &  EVALUATION 
AGENDA:  MARCH  04,  2009 


WEDNESDAY,  MARCH  4,  2009 

7:00  AM  -  5:00  PM  CONFERENCE  REGISTRATION 

7:00  AM  -  8:00  AM  CONTINENTAL  BREAKFAST  IN  DISPLAY  AREA 

8:00  AM  CALL  TO  ORDER  AND  REMARKS-(CR0WN  BALLROOM  3, 4  &  5) 

►  Mr.  Sam  Campagna,  Director,  Operations,  NDIA 

SESSION  IV:  T&E  POLICY  &  STUDIES 

Session  IV  Chair:  Dr.  Paul  Deitz,  Acting  Director,  ARL  HRED 

8:05  AM  PHYSICS  OF  FAILURE  -  THE  CRITICAL  PATH  TO  SAVING  $M  IN  T&E 

►  Dr.  W.  Forrest  Crain,  Director,  US  AMSAA 

8:35  AM  A  COMMON  PLATFORM  FOR  D0D  AND  DHS  TO  PARTNER  IN  THE  T&E  COMMUNITY 

►  Dr.  Richard  Murrow,  CEO,  Air  Academy  Associates 

9:05  AM  PREVAILING  DEVELOPMENTAL  TEST  AND  EVALUATION  (DT&E)  CHALLENGES 

►  Mr.  Thomas  Berard,  Air  Force  Flight  Test  Center,  Edwards  AFB 

9:35  AM  IMPLEMENTATION  OF  THE  RECENT  DEVELOPMENTAL  TEST  &  EVALUATION  DEFENSE  SCIENCE  BOARD  RESULTS 

►  Mr.  Christopher  DiPetto,  OUSD  (AT&T) 

10:00  AM  MORNING  BREAK  &  NETWORKING  IN  DISPLAY  AREA 


SESSION  V:  T&E  POLICY  &  STUDIES  CONTINUED 

Session  V  Chair:  VADM  Joesph  W.  Dyer,  USN  (Ret),  iRobot  Corporation 

10:30  AM  PANEL  DISCUSSION:  THE  MOVING  POLICY  TRAIN:  IMPLICATIONS 

Panelists:  Dr.  Steven  Hutchison,  DISA  TEMC 
Mr.  James,  Ruma,  GDLS 
Mr.  Brian  Simmons,  ATEC 
Dr.  Robert  Soule,  IDA 


11:15  AM  DISCUSSION:  ACQUISITION  AND  T&E:  COLLABORATING  FOR  THE  WARFIGHTER 

This  discussion  will  enable  a  fresh  examination  of  the  relative  roles  of  the  senior  Pentagon  proponents  and  the  role  of  the 
independent  oversight  functions  in  the  Pentagon  established  by  Congress  25  years  ago.  This  debate  will  include  written 
questions  submitted  from  the  audience  to  the  participants. 

Mr.  Christopher  Dipetto,  OUSD  (AT&L) 

Dr.  Ernest  Seglie,  DOT&E,  OSD 


11:45  AM  ANNUAL  TESTER  OF  THE  YEAR  AWARDS  LUNCHE0N-(CR0WN  BALLROOM  1  &  2) 

This  awards  event  is  a  highlight  of  our  annual  conference  as  it  provides  the  opportunity  to  recognize  outstanding  achievement 
in  test  and  evaluation  by  members  of  our  armed  forces,  DoD  civilians  and  DoD  contractors.  Furthermore,  what  makes  these 
awards  particularly  noteworthy  is  that  the  selections  are  made  by  the  peers  of  those  being  recognized.  Congratulations  to  all  who 
are  being  recognized  this  year! 


TEST  &  EVALUATION 
AGENDA:  MARCH  04,  2009 


SESSION  VI:  T&E  WORKFORCE  &  RESOURCES 

Session  VI  Chair:  Mr.  Charles  Larson,  SURVICE  Engineering 


1:15  PM  THE  DOD  STRATEGIC  PLAN  FOR  TEST  &  EVALUATION  RESOURCES 

►  Mr.  Jason  Coker,  OUSD  (AT&L/TRMC) 

1:45  PM  TODAY’S  AND  TOMORROW’S  DoD  TEST  &  EVALUATION  ACQUISITION  WORKFORCE 

►  Ms.  Darlene  Mosser-Kerner,  OUSD  (AT&L) 

2:15  PM  AFTERNOON  BREAK  &  NETWORKING  IN  DISPLAY  AREA 


2:45  PM  -  5:15  PM  CONCURRENT  FOCUS  AND  WORKING  DRAFT  SESSIONS 

Each  Breakout  Session  will  consist  of  both  oral  presentations  and  collaboration  between  presenters  and  audience  to  draft  the 
recommendations  and  issues  raised  during  each  session  for  presentation  to  the  plenary  session  Thursday  morning.  The  Session  host 
and  presenters  will  be  responsible  for  pulling  these  issues  together  for  later  presentation. 


SESSION  D-(CR0WN  BALLROOM  4) 

SESSION  E-(CR0WN  BALLROOM  5) 

RESOURCES,  INSTRUMENTATION,  AND  TARGETS 

APPLICATION  OF  T&E/POLICY  TO  SPECIFIC  SYSTEMS 

Chair:  Mr.  Charles  Larson , 

Chair:  RADM  Charles  “Bert” Johnston,  USN  (Ret), 

SURVICE  Engineering 

WyleLabs 

Test  and  Training  Enabling  Architecture,  TENA, 

Reliability  Growth  Planning  and  Implementation  of  the 

An  Important  Component  in  Joint  Mission  Environment 

ASA  (ALT)  Reliability  Test  Threshold 

2:45  PM 

Test  Capability  Successes 

Mr.  Gene  Hudgins,  BAE  Systems 

Dr.  J.  Brian  Hall,  Army  Evaluation  Command 

Joint  Mission  Environment  Test  Capability 

Challenges  Facing  T&E  of  Systems  to  Combat  Nuclear  and 

3:15  PM 

Radiological  Smuggling 

Mr.  Chip  Ferguson,  TRMC 

Mr.  Stephen  DeFrank,  Jr.,  Mantech  SRS ,  Inc. 

The  Department  of  Homeland  Security  Leverages  DoD 

Application  of  Legislative  Language  and  DoD  Directives  to 

3:45  PM 

Assets  for  Test  &  Evaluation 

Life  Fire  Test  &  Evaluation 

Mr.  Kerry  Wilson,  DHS  Counter- MANPADS  Program 

Mr.  James  O’Bryon,  The  OBryon  Group 

4:15  PM 

Evolving  Threats  -  How  They  Impact  T&E  Testing  and 
Infrastructure 

C-RAM  Delivers  on  Promises  and  Saves  Lives 

Mr.  Dick  Dickson,  TYBRIN  Corporation 

Mr.  James  Bloodsworth,  Army  C-RAM  Program  Directorate 

Threat/Target  Simulation  for  Live-Fire  Events 

Integrated  Display  and  Keyboard  Video  Mouse  Control 

4:45  PM 

(KNV)  of  Multiple  Computers:  A  New  Paradigm 

Mr.  Greg  Van  Splinter,  NAWCWD 

Mr.  Robert  Marcus,  RGB  Spectrum 

5: 15  PM  CONFERENCE  ADJOURNES  FOR  THE  DAY  AND  DISPLAY  AREA  CLOSED 


TEST  &  EVALUATION 
AGENDA:  MARCH  05,  2009 


THURSDAY,  MARCH  5,  2009 

7:00  AM  - 12:00  PM  CONFERENCE  REGISTRATION 

7:00  AM  -  8:00  AM  CONTINENTAL  BREAKFAST  IN  DISPLAY  AREA 

8:00  AM  CALL  TO  ORDER  AND  REMARKS-(CR0WN  BALLROOM  3, 4  &  5) 

►  Mr.  Sam  Campagna,  Director,  Operations,  NDIA 

SESSION  VII 

Session  VII  Chair:  Mr.  William  Yeakel,  ORSA  Corporation 

8:05  AM  MISSION-BASED  TEST  &  EVALUATION  STRATEGY: 

AN  INTER-AGENCY-DEVELOPED  PROCESS  TO  LINK  MISSION  CAPABILITY  WITH  SYSTEM  FUNCTIONAL  REQUIREMENTS 

►  Mr.  Christopher  Wilcox,  ATEC 

8:35  AM  EVALUATING  AN  IMMERSIVE  VIRTUAL  WORLD  FOR  ACCELERATING  T&E 

►  Mr.  Redge  Bartholomew,  Rockwell  Collins 

9:00  AM  FCS  CAPABILITIES  TO  EMPOWER  THE  SOLDIER— WHAT  IS  AND  WHAT  WILL  BE 

►  Mr.  Bud  Irish,  Vice  President,  SAIC 

9:25  AM  THE  ANALYSIS  OF  ALTERNATIVES  (A0A):  A  MISSION-ORIENTED,  EVALUATION-BASED  FRAMEWORK  FOR  DEFENSE  TEST  & 
EVALUATION 

►  Mr.  Vincent  P.  Roske,  Jr.,  IDA 

9:50  AM  T&E  PANEL  DISCUSSION:  EARLY  INVOLVEMENT 

Panel  Chair:  Dr.  Ernest  Seglie,  DOT&E 
Mr.  David  Hamilton,  AF/TE 
Dr.  James  Streilein,  ATEC 
Dr.  Stephen  Whitehead,  OPTEVFOR 


10:25  AM  MORNING  BREAK  &  NETWORKING  IN  DISPLAY  AREA 

10:50  AM  CONFERENCE  SYNOPSIS:  OUTBRIEFS  OF  BREAKOUT  /  WRITING  SESSIONS 

(Input  from  the  audience  will  also  be  entertained  during  this  session.) 

►  New  T&E  Policy  and  Policy  Implementation  —  Mr.  David  Duma,  DOT&E  and  Mr.  Robert  Wojciechowski,  APG 

►  Mission-Based  Test  &  Evaluation  —  Mr.  Britt  Bray,  DRC  and  Mr.  Jack  Sheehan,  ORSA  Corporation 

►  Improving  T&E  Efficiency  and  M&S  —  Dr.  Paul  Deitz,  ARL  and  Dr.  Mark  Kiemele,  Air  Academy  Associates 

►  System-Specific  T&E  Issues  -  Dr.  Anne  Hillegas,  ARA  Corporation  and  RADM  Charles  “Bert”  Johnston,  Wyle  Labs 

►  Resources,  Instrumentation  and  Targets,  Mr.  Dick  Dickson,  Tybrin  and  Mr.  Charles  Larson,  SURVICE  Engineering 

11:50  AM  CLOSING  REMARKS 

►  Mr.  James  O’Bryon,  Chairman,  NDIA  T&E  Division,  The  O’Bryon  Group 


12:00  PM  CONFERENCE  ADJOURNS 


TEST  &  EVALUATION 
SECONDARY AUTHORS 


SECONDARY  AUTHORS 


ABSTRACT  ID 

ABSTRACT  TITLE 

ADDITIONAL  AUTHORS 

7773 

Applying  the  SCORE  Framework 
to  Evaluate  Advanced  Military 
Technologies 

Mr.  Brian  Weiss 

7796 

System  of  Systems  -  Survivability 
Lethality  Vulnerability  Assessment 

Dr.  Jeffrey  Smith 

7817 

Extending  Test  and  Evaluation 
Modeling  and  Simulation  Capabilities 
with  Gaming  Technology 

Mr.  Robert  Bowen 

Mr.  Kevin  Van  Antwerp 

Mr.  Patrick  Heney 

Mr.  Dale  Waldrep 

7828 

GAO  Assessment  of  DOD’s  Major 
Weapon  Sytems  Acquisition  Portfolio 

Mr.  Michael  Sullivan 

7842 

ITAA  GEIA-STD-0009-- New  Best 
Practices  Standard  for  Reliability 
Assessment  and  Verification 

Dr.  Michael  Cushing 

7847 

Walking  the  Line  with  Title  10: 
Implementation  Strategies  for 
Integrated  Testing 

Ms.  Darlene  Mosser-Kerner 

7854 

The  DoD  Strategic  Plan  for  Test  and 
Evaluation  Resources 

Mr.  Chris  Paust 

7855 

Early  Decision  Analysis  Guiding 
System  Reliability  Growth 

Dr.  Donald  Gaver 

Dr.  Ernest  Seglie 

THANK  YOU  TO  OUR 
PROMOTIONAL  PARTNER 


25th  ANNUAL 
TEST  &  EVALUATION 
NATIONAL  CONFERENCE 


A  L  I  O  N 

SCIENCE  AND  TECHNOLOGY 


THANK  YOU  FOR  ATTENDING! 

WE  LOOK  FORWARD  TO 
SEEING  YOU  NEXT  YEAR 


MARCH  2-5, 2009 
SHERATON  ATLANTIC  CITY 
ATLANTIC  CITY,  NEW  JERSEY 


FEBRUARY  20-25,  2010 
SAN  DIEGO,  CALIFORNIA 
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WWW.NDIA.ORG/MEETINGS/991 0 
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Rockwell 
Collins 

Development  Scale  Problem 

•  In  large,  distributed,  complex  programs,  T&E  can  be  a 
discovery  process  rather  than  a  confirmation 

-  Many  MSLOC,  several  supplier  tiers,  many  regional  facilities 

-  Communication,  coordinating/synchronizing  supplier  actions, 
deploying  &  clarifying  information  are  problematic 

-  Iterations  of  test-analyze-fix  from  conflicting  interpretations  of 
requirements,  designs,  interfaces 

•  Collocation  optimizes  development  but  is  unlikely;  using 
conventional  media  as  a  substitute  fails 

•  Medium  that  simulates  collocation  might  mitigate  problems 

•  Internal  evaluation  focused  on  25  regional  facilities 


©  Copyright  2008  Rockwell  Collins,  Inc. 
All  rights  reserved. 
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Rockwell 
Collins 

Immersive  Virtual  Collaboration 

•  Immersive  virtual  world  simulates  collocation 

-  Includes  integrated  audio  &  visual  animation,  data  &  object 
persistence,  common  view,  integrated  development  tools 

•  Could  provide  enough  fidelity  to  collocation  for  effective 
collaboration  &  synchronizing  suppliers 

-  Strangers  meet  in  online  games  &  social  worlds,  form  teams, 
strategize,  execute  plans,  coordinate  actions 

•  Could  reduce  number  of  errors  that  escape  implementation 
phase  into  T&E 

•  Integrated  modeling  &  simulation  tools  could  provide 
environment  for  initial  T&E 


©  Copyright  2008  Rockwell  Collins,  Inc. 
All  rights  reserved. 
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Rockwell 
Collins 

Sun’s  MPK20  -  Project  Wonderland 

•  Operation  can  be  confined  within  firewall  to  resolve  third 
party  storage,  data/access  control,  identity  masking,  etc. 

•  Animation  of  configurable  avatars,  stereo  VOIP,  IM 

-  Keyboard  steers  avatar  through  prefabricated  concourse, 
conference  room,  offices,  lab 

-  Users  logged  onto  same  server  see,  hear  each  other 

-  3-D  objects  can  be  assembled  and  animated 

•  Provides  collaboration  via  3rd  party  applications  launched 
into  in-world  2-D  windows 

-  Participants  jointly  develop  artifacts  by  explicitly  sharing  control  of 
launched  applications 


©  Copyright  2008  Rockwell  Collins,  Inc. 
All  rights  reserved. 
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Rockwell 

Collins 


MPK20  -  Project  Wonderland 


©  Copyright  2008  Rockwell  Collins,  Inc. 
All  rights  reserved. 
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Rockwell 

Collins 


Code  Window,  3rd  person  view 


©  Copyright  2008  Rockwell  Collins,  Inc.  6 

All  rights  reserved. 


First  Person  View 
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Object  animation 


©  Copyright  2008  Rockwell  Collins,  Inc. 
All  rights  reserved. 
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Collaborative  design 


©  Copyright  2008  Rockwell  Collins,  Inc. 
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Rockwell 

Collins 


Code  inspection 
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Rockwell 
Collins 

Subjective  Assessment ... 

•  ...  based  on  internal  proof-of-concept  evaluation  and 
Software  Engineering  Research  Center  project 

•  Promotes  informal  communication  &  coordination  well 
beyond  IM,  tele/video-conferencing  ... 

-  ...  if  it  is  frequently  used  by  nearly  everyone 

•  Integrated  tools  improve  common  understanding,  increase 
accuracy  of  understanding  ... 

-  ...  for  requirements,  designs,  interfaces,  status  conveyed  by 
modeling/simulation  tools,  code  editors,  code  coverage  tools 

•  Scalability  yet  to  be  evaluated  . . . 

-  ...  number  of  developers,  facilities,  time  zones 


©  Copyright  2008  Rockwell  Collins,  Inc. 
All  rights  reserved. 
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Rockwell 
Collins 

Emerging  Operational  Concept 

•  Central  system  assembly  &  test  area  with  library 

•  Radial  hallways  contain  offices/labs  for  analysts, 
developers,  end-users,  acquisition  team 

•  Developers  analyze,  design,  code  in  windows  visible  to  all 
authorized  participants 

•  Library  contains  artifacts  accessed  via  integrated  tools  - 
e.g.,  modeling,  requirements  management 

-  Hold  artifact  reviews  in  library  alcoves 

-  Hold  program  reviews  in  library  auditorium 

•  Projects  spin-off  worlds  from  baseline  as  need  dictates  - 
e.g.,  unique  tools,  access  restrictions 


©  Copyright  2008  Rockwell  Collins,  Inc. 
All  rights  reserved. 
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25th  Annual  NDIA  National  T&E  Conference 

Atlantic  City  NJ 

Counter-Rocket,  Artillery,  Mortar  (C-RAM) 
Delivers  on  Promises  and  Saves  Lives 


Jim  Bloodsworth 
PD  C-RAM 
Test  Branch  Chief 
Jim.Bloodsworth@us.army.mil 
256-774-6899 


5  Mar  2009 
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Counter-Rocket,  Artillery,  Mortar  (C-RAM) 
Delivers  on  Promises  and  Saves  Lives 
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Counter-Rocket,  Artillery,  Mortar  (C-RAM) 
Delivers  on  Promises  and  Saves  Lives 


5  Mar  2009 
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Operational  Environment 


*  Indirect  fire  is  and  will  remain  a  staple  of  combat 

*  Weapons  of  choice  for  conventional  &  irregular  foes, 
worldwide: 

-  Mortars 

■  Rapidly  employed  and  used,  acceptably  accurate,  difficult  to  locate 

-  Rockets  (all  calibers  and  types) 

■  Remote  or  delayed  firing  options,  difficult  to  locate 

-  Some  possible  use  of  lighter  artillery  (75mm-105mm) 


■  More  accurate,  easier  to  move,  easier  to  locate 


Present  and  future  threats  will  exploit  vulnerabilities  of 
US/Coalition  fixed  sites,  especially  large  bases,  by 
indirect  fire  attacks 


5  Mar  2009 
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Current  Operational  Requirement 


MMMC 


HQDA  Validated  OIF 
Operational  Needs 
Statements  (ONS) 
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Operational  requirement  for  Integrated 
Force  Protection  based  on  Theater  needs: 

V  10  Sep  04  Army  Strategic  Planning  Board  Validated 
Theater  ONS  306-04  for  a  Counter  Rocket  and  Mortar 
Intercept  and  Destroy  Capability 


comm* KUiin.  Km  tQMit&ikw  ttiwtiHtm  wwww  va  ■ 

HARDEE  AYE  .  FOK3  HtTPHERSOtf  30000*1091 

OOliMAMJ^R  Hii^JOW^HfliRlk  "D«C*  £  COWIWD.  I?/1  NASOE 
FCHtTttCVHliHSflK.fi*  i  -mi 

ODAlhMWft.  UNsfiG  3  Y  AUS  asjjy  EGRCtfE.  4M>A£0«H' 

C:OW-4am<:4;A.  LH:>  JNiSf  &  SI  Ati  y  ruin  y  +'Q4f'1  SfrUfHTCSiL 

WKHm 


msm"  t» WWe».frtm,**  1 WASPS 


!  Rewire*  HCDo  oav  0AV»Q  ClO.  QftlS  2?&  :4  2-S7  34  IOC  «.  4M  M.  Ml 


MV  emtoj-t  xfTP.il  pJ^foatHYi ftonf&  mm  A5PB  *rv^:*n- 


5  jM  qppptm  imtu  waMfetu  Mi  mm*\  n  T-iWfEmi  liny  ■:■  j-h  i^i  t  tj  pr'c  rw 

♦  KJ'-  ^ rr-a  -s  '  w  JnfKnH,  mm-m.  (tti  MNI14I  *  aw 

PrTffiE.  MpjHl 


'M' .  \  \  \  \  \  \  \  \  ■ 

■ .  ■ .  ■ .  ■ .  ■ .  ■ .  ■ .  ■ .  ■ .  ■ .  ■ .  ■ .  ■ .  ■ .  ■ .  ■ .  ■ .  ■  citt 

■ .  ■.■.■.  ■.■.■.  ■.■.■.  • .  •  -m**.  Fttopi  awWw  ■ 

eft-;-; 

■SjKftteil-rC  $4r^fLWy  tf'  I  n)  X/:n y  iAuguiaftafi  ±-’j?AL*  x.'rJ 

■&$&} timid sintt.  c& 


- 1 


>  Supports  a  Mar  05  HQDA  G3 -validated 
Theater  ONS  (260-4)  for  Sensor  Interoperability 
to  Digital  Battle  Command  Systems 

>  Supports  a  Aug  05  HQDA  G3 -validated 
Theater  ONS  (05-466)  for  an  Integrated  Base 
Defense  Security  System  capability 


5  Mar  2009 
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C-RAM  Requirement  /  Background 


•  C-RAM  Requirement 

-  Jun  2004  -  Theater  submitted  Operational  Need  Statement  (ONS  306-04)  for  a  system  to  destroy 
Mortar  rounds 

-  Sep  2004  -  The  ASPB  validated  theater  ONS  and  directed  ABO  fund  C-RAM  Proof-of-Principle  Test 

-  Jan  2005  -  Results  of  C-RAM  POP  test  briefed  to  VCSA,  SecNav,  and  Dep  SecDef 

•  C-RAM  Capability 

-  Feb  2005  -  Sense  and  Warn  capability  validated  and  initial  fielding  begins 

-  Apr  2005  -  Intercept  capability  validated  and  initial  fielding  begins 

-  Jul  2005  -  IOC  of  Sense,  Warn,  and  Respond  capability  at  FOB  1 

-  Sep  05  -  C-RAM  supported  successful  attack  of  2  insurgent  mortar  teams  -  1 1  KIA,  5  captured 

-  Mar  06  -  FOC  declared  at  FOB  1  and  first  combat  intercept  (first  ever  for  a  Phalanx  system) 

*  FMS  and  Lease  Cases  approved  2QFY06 

-  May  06  -  First  Coalition  FOB  fielded 

*  IBDSoS  funded  2QFY06  and  fieldings  begin  in  support  of  C-RAM  Enhanced 
Response  capabilities  and  ONS  05-466 


5  Mar  2009 


C-RAM  Sense  and  Warn  capability  fielded  to  15  FOBs,  3  of  which 
have  Intercept  capability,  and  IBDSoS  is  installed  at  10  FOBs 


r 
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C-RAM  /  Unit  Protection 
Architecture 


Lessons  Learned 

System-of-Systems  Integration 


*  Lesson  Learned:  System-of-Systems  (SoS)  PM  must  retain  and 
execute  integration  responsibility  for  the  SoS 

-  SoS  PM  must  conduct  all  trades  in  selection  of  component  systems 

■  Avoids  corporate  biases 

-  All  DoD  Program  of  Record  (POR)  systems  should  be  assessed 

■  Avoids  service  biases 

-  SoS  PM  must  ensure  support  of  the  component  system’s  PM  for  any 
necessary  modifications 

■  SoS  PM  must  assess  risk  and  cost  for  these  modifications 

-  When  a  suitable  system  is  not  available,  selection  of  an  S&T  initiative  or  a 
COTS  item  may  be  appropriate 

■  SoS  PM  must  ensure  any  S&T  or  COTS  item  has  been  adequately  tested  and 
is  supportable 


5  Mar  2009 


The  Government  Stakeholders  Must  be  Bound  to  “Specified  Time” 
to  Avoid  Schedule  Slips  and  Cost  Increases 
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Lessons  Learned 

SoS  Capability  Development 


*  Lesson  Learned:  To  ensure  requirements  are  met,  work  with 
independent  test  agency  throughout  system  development 

-  C-RAM  Program  Office  successfully  integrated  Army,  Air  Force,  USMC,  and 
Navy  Program  of  Record  (POR)  systems  into  the  C-RAM  SoS  and  fielded  a 
SoS  that  meets  requirement  just  6  months  after  funding 

■  Required  working  closely  with  Army  Test  and  Evaluation  Command  (ATEC)  in 
all  tests,  beginning  with  the  very  first  proof-of-principle  test  in  Nov  04 
-  ATEC  has  been  a  partner  in  the  C-RAM  program 

»  They  supported  development  of  all  test  and  data  collection  plans  to  ensure  adequate 
data  would  be  available  to  prepare  a  Capabilities  and  Limitations  report 

»  No  modifications  were  fielded  without  an  ATEC  Capabilities  and  Limitations  Report 


5  Mar  2009 


C-RAM  SoS  Capability  Development  Approach  Enabled  Accelerated  Fielding 
and  Continuous  Improvement  of  a  Force  Multiplier  in  a  Combat  Zone 
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Lessons  Learned 

SoS  Capability  Development 


•  Lesson  Learned:  To  reduce  risk  and  ensure  supportability,  fund 
component  system  PORs  to  implement  all  changes  to  their  systems. 
-  Multiple  changes  were  required  in  the  C-RAM’s  POR  component  systems 

■  All  such  changes  were  agreed  to  between  the  C-RAM  and  POR  PMs; 
developed  by  the  POR  PMs;  funded  by  C-RAM;  and  then  jointly  tested 

■  When  there  were  conflicts  between  the  POR  system’s  requirements  and  C- 
RAM  requirements,  the  Combat  Developer  helped  define  courses  of  action 
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C-RAM  SoS  Capability  Development  Approach  Enabled  Accelerated  Fielding 
and  Continuous  Improvement  of  a  Force  Multiplier  in  a  Combat  Zone 


Lessons  Learned 

Training 


*  Lesson  Learned.  SoS  PMs  need  to  focus  on  Collective  Training 

-  Development  of  CONOPs,  TTPs,  and  crew  drills 

■  PM  C-RAM  provided  Fires  Center  of  Excellence  (COE)  tactical  equipment  to  enable 
development  of  these  essential  operational  procedures 

-  Individual  Training 

■  Provided  equipment  to  Fires  COE  and  trained-the-trainers  to  enable  individual  training 

■  Supported  annual  Mobile  Training  Team  visits  to  theater  to  assess  training  and  identify 
requirements  for  changes  in  both  training  and  CONOPS/TTP 

-  Collective  Training 

■  Provided  Fires  COE  tactical  suite  of  equipment  to  support  collective  training  of 
deploying  C-RAM  units  and  supported  conduct  of  their  Mission  Readiness  Exercises 

■  Fielded  C-RAM  equipment  to  Army  and  USMC  Combat  Training  Centers  (CTCs)  to 
enable  training  of  units  rotating  to  theater 

■  Developed  stimulation  capability  to  support  on-site  unit  training  to  maintain  proficiency 
and  to  train  for  new  threats 

-  Warfighter  Feedback 

■  PM  C-RAM  conducts  weekly  Secure  VTCs  with  theater  to  provide  Warfighter  a  direct 
link  to  C-RAM  staff,  the  combat  developer,  and  PMs  of  other  component  systems 
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Close  Coordination  with  Combat  Developer  Has  Ensured  C-RAM  Units  are 
Prepared  for  Combat,  and  Identified  Additional  Changes  Required  in  C-RAM 
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Lessons  Learned 

Responsive  Support  for  the  Warfighter 


*  Lesson  Learned.  PMs  must  plan  for  and  be  prepared  to  support  urgent 
Warfighter  changes 

-  “No  plan  survives  first  contact” 

-  C-RAM  provides  a  personal  PM  representative  on  MNC-I  staff  to  coordinate 
and  respond  to: 

■  Multiple  changes  in  fielding  locations  and  priorities 

■  Requirement  to  relocate  /  reinstall  C-RAM  capability  from  one  FOB  to  another 

■  Changes  in  threat  tactics 

■  Expansion  of  capabilities  to  support  tracking  of  insurgent  crews  by  integration 
and  cuing  of  EO/IR  sensors 

■  Additional  available  sensors  at  FOBs  by  integrating  them  (UTAMS,  Shadow) 

*  Lesson  Learned.  PMs  need  to  include  collection  and  analysis  of  data  on 
system  performance  concurrent  with  their  fielding  to  enable  rapid 
identification  and  localization  of  performance  issues 

-  C-RAM  provides  Warfighter: 

■  24/7  forensics  analysis  team  to  analyze  ALL  events  -  successes  and  failures 

-  Analysis  is  normally  completed  within  24  hours  for  major  events 

■  Secure  commercial  SATCOM  links  back  to  C-RAM  Program  Office  in  Huntsville 
to  enable  immediate  transmission  of  all  classified  forensics  data 


PM’s  and  TRADOC  Should  be  Funded  to  do  this  in  Peace-Time 


5  Mar  2009 
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Systems  Engineering  with  Embedded 
Forensics  Process  (U) 


5  Mar  2009 
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Forensic  Data  Process  (U) 


5  Mar  2009 
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SUMMARY 


•  Through  judicious  out-of-the-box  thinking  from 
the  gitgo,  PD  C-RAM  has  taken  a  real-time,  current 
warfighter  threat  and  turned  it  into  a  long-term 
solution  that  addresses  not  only  today’s  tactical 
theater  of  operations,  but  many  diverse  ones  in  the 
future. 

•  The  Counter-Rocket,  Artillery,  Mortar  Program: 
continuing  to  deliver  on  its  promises  and  saving 
warfighter  lives  TODAY. 


5  Mar  2009 
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The  DoD  Strategic  Plan  for  Test 
and  Evaluation  Resources 


Mr,  Jason  Coker 
Mr,  Christopher  Paust 

March  4, 2009 


Test  Resource  Management  Center 
1225  South  Clark  Street,  Arlington  VA 

22202 


Outline 


•  TRMC  Background 

•  TRMC  Mission  &  Roles 

•  Organizational  Structure 

•  Strategic  Planning  Purpose 

•  Strategic  Plan  Publication  Date 
Realignment 

•  Strategic  Plan  Development  Process 

-  Influence  on  CTEIP,  T&E  S&T, 
Service/Agency  Investments 

-  Working  Groups  and  the  Reliance  Process 

•  Summary 


TRMC  Background 

•  The  December  2000  Defense  Science  Board’s 
review  of  DoD’s  major  range  and  test  facilities 
identified  a  number  of  issues  regarding  inadequate 
funding,  management,  and  support  of  these  test 
ranges  and  facilities 

•  To  address  these  issues,  Congress,  via  the  2003 
NDAA,  directed: 

-  the  Services  to  fully  fund  the  DoD  Test  Range  institutional 
cost 

-  the  SecDef  to  establish  a  DoD-level  resource  management 
organization 

•  DoD  Directive  (DoDD)  5105.71  established  the 
TRMC  as  a  DoD  field  activity  under  the  authority, 
direction,  and  control  of  the  USD(AT&L) 
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TRMC  Mission,  Vision,  and  Goal 


DoDD  5105.71 


•  Mission:  "Plan  for  and  assess  the  adequacy  of  the. . . 
MRTFB...[and]  to  provide  adequate  testing  in  support  of 
development,  acquisition,  fielding,  and  sustainment  of 
defense  systems;  and,  maintain  awareness  of  other  T&E 
facilities  and  resources,  within  and  outside  the 
Department,  and  their  impacts  on  DoD  requirements." 

•  Vision:  The  DoD  T&E  ranges  and  facilities  will  be  fully 
capable  of  supporting  the  Department  with  quality 
products  and  services  in  a  responsive  and  affordable 
manner. 

•  Goal:  Robust  and  flexible  T&E  capabilities  to  support  the 
Warfighter. 


Roles  and  Missions  of  the  TRMC 


UNDER  SECRETARY  OF  DEFENSE 
(ACQUISITION,  TECHNOLOGY  AND  LOGISTICS) 


TRMC  Organization 
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Strategic  Planning  Purpose 


•  Provides  a  vision  of  the  capabilities  and  investments 
needed  to  support  the  testing  of  future  weapons  systems 

•  Provides  guidance  for  the  planning,  programming, 
budgeting  and  execution  of  T&E  resources 

-  Influences  Services  T&E  POM  Investments  through  the  DoD  T&E 
Budget  Certification  process 

•  The  strategic  plan  must  eventually  consider  all  available 
T&E  capabilities: 

-  DoD  T&E  (MRTFB) 

-  Governmental  non-DoD 

-  Commercial 

-  International 
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Strategic  Planning  Purpose  (con’t) 

•  Near-term  plans  will  focus  on  MRTFB  and  migrate  to 
addressing  the  broader  spectrum  of  T&E 

•  Drive  the  Services  Needs  and  Solutions  Reliance  process 

•  Guides  TRMC  Science  and  Technology  (S&T)  and  Central 
Test  and  Evaluation  Investment  Program  (CTEIP) 
investments  to  support  DoD  T&E  Strategic  Plan  gaps 


9 


Strategic  Planning  Looks  at  ALL 

T&E  Resources 


T&E  Resources:  A  collective  term  that  encompasses  the 
requisite  Workforce,  Infrastructure  and  Funding  resulting 
in  a  T&E  Capability,  by  means  of  the  T&E  Processes 


Major  Range  and  Test  Facilities  Base 

(MRTFB) 


ttevil  Air  Warfare  Center  ■  Weapon*  Oitlsiwi, 
China  Lake,  CA  (USN1' 


TEST  FACILITIES 


DoD  Strategic  Plan  for  T&E  Resources 

and  the  PPBE  Cycle 
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Advantages  for  Realignment 


•  Supports  OSD,  Services,  and  Agencies 
PPBES  and  POM  processes 

•  Supports  Services  and  Agencies  Reliance 
Needs  and  Solutions  Process 

-  Established  MOA 

•  Brings  a  purple  flavor  to  the  process 

•  Provides  a  top  down  perspective  to  the 
historically  bottoms  up  process 

•  Synergy  through  aligned  investments 


tttttttttttttt 


The  Strategic  Planning  Process 


Review  Service/OSD 
Strategic  Plans 


TARA  Reviews 


Review  ACTD  Master 
Plan 


Focus  Area  Working 
Groups 


T&E/S&T  Roadmaps 


Review  DARPA 
Strategic  Plan 


T&E  Workshops, 
Conferences,  and 
Symposia 


Range  Visits 


AG  — 

Reliance  Panel 

BES  — 

Participation 

CPR  — 

Range  Visits 

CP 

FG 

T&E  Workshops, 
Conferences,  and 

IPL  — 

Symposia 

IP&B 

JPG 

Review  Service  T&E 

Roadmaps 

N&S 

PB  — ^ 

PEO  Discussions 

P&BREV  — ^ 

PBD 

TEMP  Reviews 

PDM  — 

T1  Reviews 

POM 

RACS  — 

R-2  Reviews 

w  _ ^ 

Strategic 

Tactical 

Assessment  of  T&E 

Assessment  of  T&E 

Requirements 

Requirements  4 

INVESTMENTS 


Service  POM 
CTEIP 
T&E  S&T 


TOrrent  T&E 

Capabilities 

Database 
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Building  the  Strategic  Plan 


Top-Down  Inputs 


Requirements 

Assessment 

& 

Analysis 


l&M 

Reviews 


S&T 

TARA 

Reviews 


MRTFB 

Annual 


Bottom-up  Inputs 


Directed  Energy 
Nuclear  Weapons  Effects 
Hypersonics 

Distributed  Test/Cyber/IO 
Urban  Environments 
Unmanned  &  Autonomous 
Systems/Maritime  Surveillance 
IED  Defeat 
Biometrics 
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Strategic  Planning  Interrelation  and  Roles 


Departmental 

Guidance 

f 

Acquisition  Program 
Requirements 


vond  «1©  Fy 

Areas  of  Inf»r^ 

Biometrics 

Maritime  Surveillance 
Urban  Envionmerts 


/  Embedded  Instrumentation 
/  Sp  e  drum  EM  c  I  end  T eehm  dogy 
Unmanned  and  Autonomous  Systei 
Uypersonlcs 
Mulli- spectral 
Directed  Energy  > 
Net  Centric  Systems 


CTEIR 

Investment 

Process 


Sposj  Tert  Gapabilty 
Interoperability  ■ 
Nudear  Weapons  Effects 

Courrter-IED 

Manpower 


Service 
Needs  and 
Solutions 
Process 


Capability  Aye_  .s 

^  Land  Combat  ^ 

Air  Combat 
Sea  Combat 
Space  Combat 
Electronic  Combat 
C4ISR 

Armaments  and  Munitions 
Targets  and  Threats 
Common  Range  Instrumentation 
Test  Environments 


T&E 

Capability 


Current  Capability 
Assessments 


Capability 

Database 
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T&E  Investments  and  the 
Strategic  Planning  Process 


Operational 

Requirements 


Acquisition 

Program 

Suite 


v 


T&E 

Requirements 


Strategic  Plan  Focus  Area 
Selection  Process 


Evaluate 
Operational 
Needs  and 
High  Level 
Guidance 


Finalize  FA  List 
for  Current 
Development 
Cycle 
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Strategic  Planning  Focus  Area 
Working  Groups 


•  Focus  Area  Working  Group  Composition 

-  TRMC  Lead 

-  Lead  POC  form  each  Service/Agency  Headquarters 

-  Service/Agency  Subject  Matter  Experts  for  each  Technical 
Area  (Action  Officer) 

-  Multiple  Supporting  Members 

•  Headquarters  POCs  Serve  as  Final  Authority 

•  Action  Officers 

-  Serves  as  principal  POC  between  TRMC  and  their 
Service/Agency 

-  Reviews  and  vets  focus  area  development  throughout  their 
respective  Service/Agency 

-  Participates  in  meetings  as  established  by  the  TRMC  Focus 
Area  leads 
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Summary 


•  Strategic  Plan  provides  fundamental 
support  to  the  TRMC  mission 

•  Foundational  document  for  DoD  T&E 
resources 

•  Must  be  collaboratively  developed  with  the 
Services  and  Agencies  it  influences 

•  Influence’s  T&E  S&T,  CTEIP,  and 
Service/Agency  investments 


Questions? 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


Physics  of  Failure  -  The  Critical  Path  to  Saving  $M  in  T  &  E 


Dr.  W.  Forrest  Crain 
410-278-6614 
DSN  298-6614 


Presentation  approved  for  public  release;  distribution  is  unlimited. 


RDECOM 


Agenda 


□  Cost  of  Failure 

□  Background 
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What  is  the  Cost  of  Failure? 


Type  of  Business 


Lost  Revenue  per  Hour 


Retail  Brokerages  $6,450,000 

Credit  Card  Sales  Authorization  $2,600,000 

Home  Shopping  Channels  $113,750 

Catalog  Sales  Centers  $90,000 

Airline  Reservation  Centers  $89,500 

Cellular  Service  Activations  $4 1 ,000 

Package  Shipping  Services  $28,250 

Online  Network  Connect  Fees  $22,250 

ATM  Service  Fees  $14,500 

Supermarkets  $  1 0,000 


continued 


source:  “Ready  when  chips,  lines  are  down;  Firm  offers  clients  work  space  in  crises,”  Margaret  Webb  Pressler, 

Washington  Post,  Washington,  Dec.  18,  1999,  pg.  El  &  U  of  MD  CALCE  Center 
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What  is  the  Cost  of  Failure? 


LAPTOP  LAWSUIT* 


Major  corporation  agreed  to  MISSION  FAILURE 

$2. 1  Billion  Settlement  for  selling  AND/OR  LOSS  OF  LIFE 
allegedly  defective  laptops. 

AN  AUTOMOTIVE  RECALL 

Dear _ Customer,  This  notice  is  sent  to  you 

in  accordance  with  the  requirements  of  the 

National  Traffic  and  Motor  Safety  Act.  _ has 

decided  that  a  defect  which  relates  to  motor  vehicle 

safety  exists  in  certain _ vehicles.  ... 

Windshield  wiper  motors  may  fail  after  a  year 
or  more...  as  a  result  of  cracked  solder  joints  on 

.  77  .  .  7  7  *  Wall  Street  Journal,  1  Nov  99  pg.l 

the  controller  circuit  board. 
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Background 


PoF  - 


A  Comprehensive  Engineering  Based  Reliability  Approach 

>  Also  termed  “Physics  of  Failure”  (PoF), 
“Predictive  Technology”,  “Predictive 
Engineering”,  “Physics  of  Reliability”  and 
models  the  root  causes  of  failure  that  include 
fatigue,  fracture,  corrosion,  and  wear. 


Stress  (e.g., 
vibration)  is 
propagated  from 
the  system  level 
to  a  failure  site 


>  Industry,  academy,  and  government  develop 
failure  models  and  CAD  tools  that  address 
specific  materials,  failure  sites,  and  design 
architectures. 


Benefits 

•Influence  design  early 
•Eliminate  failures  prior  to  test 
•Increased  chance  of  passing 
test 

•Enhanced  fielded  reliability 
•Improved  prognostics 
•Decreased  O&S  costs 


Failure  root- 
cause  is  cracking 
of  solder  joint 
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Test  &  Evaluation  Environment 


m  1  U  PI  m  T  m  1 


□  May  not  be  enough  time  in  schedule  for  desired  test 

□  Funding  may  be  insufficient 

□  Test  asset  availability  limited;  assets  may  be  expensive, 
scarce,  or  needed  for  the  war  effort 

□  T&E  IPT  process  highly  competitive  as  proponents  for  each 
element’s  evaluation  push  to  ensure  that  their  data 
requirements  will  be  met 

□  Testers  and  evaluators  in  a  difficult  position  -  under 
pressure  to  make  do... 

□  Need  to  include  new  approaches  that  leverage  test 
activities  to  provide  more  information 


Need  to  get  the  most  from  every  test! 
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rdecom^  Problem  Statement  and  Benefits 


□  Problem  Statement:  The  Army  requires  Physics  of 
Failure  (PoF)  throughout  the  materiel  acquisition  life 
cycle  to  mitigate  current  cost,  schedule,  and 
performance  issues. 

□  Benefits 

•  PoF,  relying  on  physics-based  analysis,  provides  Testers, 
Evaluators,  and  Program  Managers  the  ability  to  field 
highly-reliable  materiel. 

•  PoF  ensures  that  Testers,  Evaluators,  and  Program 
Managers  get  the  most  out  of  every  test. 
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What  is  AMSAA  Doing  with  PoF? 


WxlSm 


□  Supporting  T&E  and  acquisition  communities  with  Physics-of-Failure  (PoF)  analysis 

•  System-level  dynamics  models,  component  finite  element  models,  fatigue-life  models 
■  Reveals  the  underlying  physics  of  the  hardware  in  its  mission  environment 


Outputs  include: 

S  Forces  acting  on  a  system 
S  Displacements  of  components 
S  Accelerations 
s  Stress  levels 
S  Weak  points  of  the  design 


PoF  enables  the  Army  to... 

S Anticipate  &  understand  test 
performance 

S Not  be  surprised  by  test 
performance 


•  What  can  the  Army  do  with  these  new  insights? 

■  Work  with  ATEC,  PMs,  RDECs,  &  contractors  to  fix  the  components  responsible 
for  poor  system  reliability 

■  Make  meaningful  and  helpful  suggestions  for  corrective  action  approaches 

■  Encourage  contractors  to  use  the  best  analysis  tools 

■  Provide  peer-level  review  of  contractor  designs  and  suggestions 

■  Reap  theibenefits  of  increased  customer  knowledge 


Increased  1 

A  Increased 

Lower  Life- 

Better 

Higher 

Knowledge  | 

^Warfighter  Benefit 

Cycle  Cost 

Performance 

Reliability 

TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

8 


rdecomT^ 

Physics-of-Failure  Process 

for  Electronics 

ID 


Support  for  importing 
CAD  design  files 


Toolbox 


Product  Modeling  and  Databases 
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What  is  AMSAA  doing  with  the  Reliability 

Scorecard? 


M  Vairiri 

AMSAA 

EjKfvNtfeCi-  m 


□ 


□ 


8  Scorecard  Categories  \- 

•  Reliability  requirements  and  | 

planning  oj 

•  Training  and  development 

•  Reliability  analysis  < 

0£ 

•  Reliability  testing  ^ 

Q£ 

•  Supply  chain  management  ° 

•  Failure  tracking  and  reporting 

•  Verification  and  validation 

•  Reliability  improvements 

For  each  Scorecard  Category 
there  are  several  elements  with 
associated  rating  criteria 


Identify  weak  performers 
early  using  this  structured 
and  analytic  approach 


One  example  element  from  the  Reliability 
Analysis  Scorecard  Category 


Critical  loads  and  stresses  are 
characterized;  life  cycle  environment 
and  operation  duty  cycle  stresses  are 
characterized 

Clearly  define  estimates  of  life-cycle  user 
and  environmental  loads,  update 
periodically,  verify  with  measurements  on 
pre-production  systems/products.  The 
developer  must  characterize  the  critical 
loads  and  stresses.  Validate  with  additional 
testing  and  data  collection. 

Green 

Estimate  life-cycle  user  environmental  loads 
from  "like-systems"  in  similar  operational 
environments.  Measurements  not  verified 
on  actual  system  through  testing  and  data 
collection. 

Yellow 

Life-cycle  user  environmental  loads  and 
duty  cycle  stresses  are  not  defined. 

Red 

>=> 


Encourages  use  of  Physics- 
of-Failure  analysis 
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AMSAA  Reliability  Program  Scorecard 

Tools  fl  Forms 


Ranking 


W6  of  36  items 
How  is  ranking 

calculated? 


User  Reviews 

o 


^  Activity 


Assess  reliability 
before  testing 


Long  Description 


Tit  is  f\flS  Excel -based  tool  may  be  used  to  qualitatively  score  tbe  elements  of  a  Reliability  Program . 
The  covered  elements  are: 


1 .  Reliability  Requirements  and  Planning 

2.  Training  and  Development 

3.  Reliability  Analysis 

4.  Reliability  Testing 

5.  Supply  Chain  Management 

6.  Failure  Tracking  and  Reporting 

7.  Verification  and  Validation 


l^|  40  Internet 
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jx  -  Micros.., 
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Just  a  Few  of  the  PoF  Success  Stories 


Surveillance  System 

•  Analysis  showed 
commercial  ciruit 
card  OK 


$1.2M  Saved 


$27M  Cost 


Power  Supply  Reliability  Improved 


•  Significant  failures 
reduced  with 
minimal  cost  fix 


Tri-Service  Radio  Avoidance 


•  Identified  weak  link  in 
design 


Army  Vehicle 


S500K 

Cost  Avoidance 


Mobile  Bridge 

•  Reduced 
testing 


$1.5M  Savings 


•  Reduced  testing 
through  M&S 


Tactical  Receiver 

•  Reliability  design 
enhancements  incorporated 


New  Missile  System 

•  PoF  analysis  on  Plastic  Ball 
Grid  Array 

Evaluate  New 
Technologies 
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Identified  Challenges 


□  Lead-free  electronics 

□  Counterfeit  parts 

□  No  Fault  Founds 

□  Still  a  lack  of  routine  application  of 
electronics  Physics-of-Failure  analysis 
early  in  the  development  process 

□  Continued  use  of  MIL-HDBK-217  instead 
of  rigorous  engineering  practices 
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Summary 


AMSAA 

Eifc’lIvNCT  ir» 


□  PoF,  relying  on  physics-based  analysis, 
Drovides  Testers,  Evaluators,  and  Program 
Managers  the  ability  to  field  highly-reliable 
materiel. 


□  PoF  ensures  that  Testers,  Evaluators,  and 
Program  Managers  get  the  most  out  of  every 
test. 


□  PoF  analysis  tools  can  significantly  contribute 
today. 
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International  Corporation  ® 

Leading  the  Convergence  of  National  Security  and  Technology5" 


Systems  Engineering 
and  Integration 


Information 

Technology 


Intelligence  Analysis 
and  Mission 
Operations 


Global  Integrated 
Logistics  Systems 
and  Support 
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NDIA  25th  Annual  T&E  National  Conference 


Challenges  Facing  T&E 
Of  Systems  To 
Combat  Rad/Nuc  Smuggling 


Stephen  M.  DeFrank  Jr. 

Senior  Test  Engineer 
ManTeeh  SRS  Technologies  Inc. 
stephen.defrank@mantech.com 
703-907-3645 

March  4,  2009 
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Who  is  ManTech? 


•  The  ManTech  culture  is  based  on  programmatic  and 
technical  excellence,  mission  support,  quality,  trust, 
integrity  and  ethics 

•  7,000+  highly  skilled  employees 

-  75%  with  security  clearances 


The  ManTech  Impact: 

-  A  culture  the  emphasizes  practical 
implementation  program  management 
and  systems  engineering  processes 

-  We  create  added  value  through 
quality,  innovation,  and  partnership 


-  Approximately  50%  cleared  Top  Secret  and  above 

180  locations  world  wide  with  operations  in  42  countries 
and  39  states 


ITALY 

Coltano 

Vicenza 

Camp  Darby,  Livorno 


GERMANY 

Kaiserslautern 

Bad  Aibling 

Mannheim 

Bitburg 

Seckenheim 

Darmstadt 

Stuttgart 

Friedrichsfelde 

WUerzbu-g 

Heideberg 

Augsburg 

-  World  class  engineers  and  technical 
specialists 

-  Unique  experience  in  transforming 
leading  edge  concepts  and 
technologies  into  operationally 
effective  systems 


Operational  Conflict  of  Interest 
(OCI)  Free 

-  We  provide  our  customers  with  the 
right  information  at  the  right  time  to 
make  informed  programmatic  and 
technical  decisions 

-  We  don’t  make  things,  we  provide  the 
environment  to  make  them  better 

-  Our  people  are  our  only  product 


Our  core  competencies  cross  many  domains  and  are  attentive  to  our 
customer’s  technical  and  programmatic  support  needs 


Homeland  Security 
WMD  Detection 

POE  Security 

i 

Unattended  Aircraft  Systems 
Electro-optics  and  lasers 

Radar  and  Signal  Processing 

Aircraft  protection 

C4ISR 

Information  Technology 

L 

l—h 

HIP 

Program  Management/Systems  Acquisition 
-Acquisition  strategy  development 

-  Program  planning  and  control/EVM 

-  Budget  and  financial  management  support 

-  Milestone  and  technical  reviews 
Systems  Engineering  and  Integration 

-Complex  system  of  systems  engineering 

-  Enterprise  architecture 
-CONOPS 

-  Requirements  analysis  and  management 

-  Design/development  oversight  and  review 
-Modeling,  Simulation  and  Analysis 
-Test  and  evaluation/IV&V 

Total  Life-Cycle  Sustainment 

-  Life-cycle  assessment/business  case  analysis 
-ILSPs 

-  Design  for  sustainment 

-  Reliability,  supportability,  maintainability 


•  Environmental  Planning  Services 

-NEPA  planning,  compliance,  and  assessment 

-  EIS,  EA,  environmental  studies 

-  Endangered  species  act 

•  Mission  Assurance/Specialty  Engineering 

-  System  safety 

-  Reliability  and  quality  engineering 
-Software  assurance 

-  Software  IV&V 
-Risk  management 

•  Test  &  Evaluation 

-Test  Planning 
-Test  Execution 
-Analysis  and  Reporting 

•  Program  Protection/System  Assurance 
Cyber  Security 

-Operational  environment 

-  Critical  information 
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Background 


*  End  of  the  Cold  War  Era  has  given 
way  to  the  War  on  Terrorism 

*  Has  Caused  a  Shift  in  Focus  From 
Nuclear  Deterrence  To  Nuclear 
Detection 

*  April  15,  2005  President 
Established  the  Domestic  Nuclear 
Detection  Office  (DNDO)  -  Under 
DHS 

*  ManTeeh  SRS  Selected  as  Systems 
Engineering  Support  Program 


DNDO  Mission  &  Scope  of  SESP 


•  Mission  of  DNDO 

*  DNDO  is  a  jointly-staffed,  national  office 
founded  on  April  15,  2005,  to  improve  the 
Nation’s  capability  to  detect  and  report 
unauthorized  attempts  to  import,  possess, 
store,  develop,  or  transport  nuclear  or 
radiological  material  for  use  against  the 
Nation,  and  to  further  enhance  this 
capability  over  time 

•  Scope  of  the  SESP 

•  Provide  systems  engineering  support  for: 

•  Global  nuclear  detection  and  reporting 
architecture 

•  User  needs  and  requirements  development 

•  Program  Management  and  Acquisition 
support 

•  Systems  Engineering  Process  Development 

•  Detection  System  design,  integration  and 
test 
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DNDO  Mission  Challenge 


Airport 

inspection 


Federal,  state,  and/locai 
o  Maw  enforcement 


Interior-modal 

detection 


Land  portal  monitoring 

1111  um 


Border  Patro 


interdiction 


Seaport 

monitoring 


Perimeter 

protection 


Coast  Guard 
interdiction 


— ►  Major  seaport 
— ►  Land  border  crossing 
©  Major  international  airport 
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DNDO  Challenges  Executing  Rad/Nuc  Testing 


Challenge 

Solution 

0  No  T&E  Rad/Nuc  Executing  Agent 

0  Combined  Test  Team 

*  Field  systems  quickly 

0  Rapid-cycle  test  framework 

*  Lack  of  detection  system  standards 

*  Standards  Program 
“GRaDER” 

0  Access  to  high  fidelity  sources 

•  RNCTEC  Facility 

•  Rail  Test  Center 

0  Maritime  Test  Bed 

0  Operationally  relevant  testing 

0  Threats  in  operational  environment 

ManTeeh 
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No  T&E  Rad/Nuc  Executing  Agent 


•  DNDO  has  created  a  Combined  Test  Team  to  meet  the 
Rad/Nuc  T&E  challenge 

*  Combined  Test  Team: 

-  DHS  Organizations  (DNDO,  FEMA  and  CBP) 

-  National  Institute  of  Standards  (NIST) 

-  John  Hopkins  University/Applied  Physics  Lab  (JHU/APL) 

-  DOE  Labs  (Savannah  River,  Los  Alamos,  Pacific  NW  Lab)) 

-  Nevada  Test  Site  (NTS) 

-  Industry  (ManTeeh) 


ManTech 

International  Corporation  * 


Company  Proprietary 


Field  Solutions  Quickly 


Test  Documentation  (DNDO  01-1) 

*  Test  Planning  Phase 

-  Test  Management  Plan 

-  Test  &  Evaluation  Master  Plans  (TEMPS) 

-  Test  Plans 

-  Analysis  Plans 

-  Data  Collection  &  Validation  Plans 

•Test  Execution  Phase 

-  Test  Procedures 

-  Real  Time  Data  Validation 

•  Test  Reporting  Phase 

-  Data  Analysis 

-  Reporting  (Tech/Peer  Reviews) 


DNDO 

01-1 

T&E 

Milestones 


SESP 

T&E 

Templates 

(Tailorable) 


CONTINUOUS  EVALUATION  OF  TEST  EXECUTION  RESULTS  AGAINST 
EACH  PHASE  TO  INCREASE  EFFICIENCY  AND  EFFECTIVENESS 
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Lack  of  Detection  System  Standards-  GRaDER  Program 


Mission  for  Graduated  Radiological/Nuclear  Detector  Evaluation  &  Reporting  Program 

*  Identify  radiation  detection  products  that  satisfy  standards  and 
DHS  Mission 


Testing  Evaluation  Procurement 


Objectives: 

*  Provide  infrastructure  for  the  collection  of  high  integrity  test 
data 

*  Standardize  instrument  testing  and  presentation  of  test  results 
to  assure  valid  comparisons  and  easily  interpreted  results 
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Operationally  Relevant  Testing 


•  Radiation/Nuclear  Countermeasures  T&E  Complex 

-  Location:  Nevada  Test  Site 

-  Capabilities:  Test  with  high  fidelity  threat  sources 

•  Maritime  Test  Bed 

-  Location:  Savannah  Rivers  National  Lab 

-  Capabilities:  Test  small  maritime  craft  in  operational  like  conditions 

•  Rail  Test  Center 

-  Location:  Tacoma,  WA 

-  Capabilities:  Test  intermodal  rail  without  impeding  flow  of  commerce 


These  Test  Facilities  Will  Allow  the  Use  of  High  Fidelity  Targets  in 
Operationally  Relevant  Locations  Without  Disrupting  the  Flow  of  Commerce 
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Successful  DNDO  Test  Campaigns 


Advanced  Spectroscopic  Portal  (ASP)  Test  Campaign  (Jan-Feb  07 

&  Jul-Aug  08) 

—  Purpose:  DNDO  test  to  support  the  DHS  Secretarial 
certification  for  ASP  Deployment  and  to  support  algorithm 
development 

—  Description:  Test  of  ASP  prototype  systems  alongside 
currently  deployed  PVT  system  against  high  fidelity  threat 
objects  in  various  configurations. 


•  ASP  New  York  Container  Terminal  (NYCT)  Test  Campaign  (Jan-Feb  07) 


—  Purpose:  DNDO  test  to  support  the  DHS  Secretarial 
certification  for  ASP  Deployment  and  to  support 
algorithm  development 

—  Description:  Test  of  three  ASP  prototype  systems  in  a 
stream  of  commerce  at  NYCT 
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Successful  DNDO  Test  Campaigns  (continued) 


•  International  General  Aviation  (IGA)  Test  Campaign  (Feb  -  Jun  08) 

—  Purpose:  DNDO  test  to  support  the  IGA  radiation  detection 
program 

—  Description:  A  test  of  the  operational  performance  of  the 
currently  deployed  GR  135  radiation  detectors  and  possible 
alternatives  in  standard  operating  procedures  and  technologies 
against  three  sizes  of  aircraft 

*  Human  Portable  Radiation  Detector  System  Test  Campaign  (Jan 

—  Purpose:  To  assess  the  radiological/nuclear  performance  of 
HPRDS  and  COTS  systems 

—  Description:  Test  of  over  20  COTS/HPRDS  handheld, 
backpack,  and  mobile  detection  systems  in  experimental  non- 
operationally  relevant  scenarios  to  gather  instrument 
performance  independent  of  the  user 


-  Feb  08) 
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Successful  DNDO  Test  Campaigns  (continued) 


•  Anole  Test  Campaign  (Jan-Feb  2006) 


—  Purpose:  Testing  of  Portable  and  Mobile  Radiation  detectors  to 
support  State  and  Local  law  enforcement  procurement  decisions 

—  Description:  Tested  over  30  COTS/GOTS  handheld,  backpack, 
and  mobile  detection  systems  in  three  operationally  relevant 
scenarios  which  included  screening,  sweeping  and 
portal/chokepoint  operations 


Bobcat  Test  Campaign  (Jui-Aug  2006) 

—  Purpose:  Testing  of  Personal  Radiation  Detectors  (PRDS)  to 
support  State  and  Local  law  enforcement  procurement  decisions 

—  Description:  Tested  over  30  PRDS  in  four  operationally  relevant 
scenarios,  which  included,  pedestrian  sweeping,  screening, 
portal/chokepoint  operations,  and  mobile  sweeping 


Crawdad  Test  Campaign  (Jui-Aug  2008) 

—  Purpose:  Testing  of  Boat  Mounted  Radiation  Detectors  to 
support  State  and  Local  law  enforcement  procurement  decisions 
and  determine  requirements  for  future  systems 

—  Description:  Tested  1 1  COTS  &  GOTS  detectors  in  four 
operationally  relevant  scenarios  at  the  Savannah  River  National 
Labs  L-Lake 
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Summary 


*  Major  national  concern  is  potential  Rad/Nuc  terrorist  attack 

*  DHS/DNDO  is  charged  with  improving  nation’s  capability  to 
detect  and  report  such  terrorist  attempts 

0  DNDO  is  meeting  the  T&E  challenges  through  formation  of  a 
combined  test  team  and  establishing  rapid-cycle  test  framework 

*  The  DNDO  GRaDER  program  will  evaluate  detection  systems 
and  begin  to  standardize  the  radiation  detection  industry 

*  DNDO  has  established  test  facilities  for  various  venues  that  has 
begun  to  make  operational  testing  possible 


Proven  T&E  processes  coupled  with  advanced  test  ranges  will  help  DNDO 
ensure  operationally  effective  and  suitable  systems  are  deployed 
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Evolving  Threats 

“How  they  Impact  T&E  Testing  and  Infrastructure 

By 


Mr.  Dick  Dickson 

GPS  Based  Range  Instrumentation  and  Equipment  IPT  Lead 

TYBRIN  Corporation 


Presented  at 

NATIONAL  DEFENSE  INDUSTRIAL  ASSOCIATION 
National  Test  &  Evaluation  Conference 


March  2009 
Atlantic  City,  NJ 


Background 
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If  their  economy  is  destroyed,  they  will  be  busy  with  their 
own  affairs  rather  than  enslaving  the  weak  peoples.  It  is 
very  important  to  concentrate  on  hitting  the  U.S.  economy 
through  all  possible  means. 

The  young  men  [of  the  Jihad]  need  to  seek  out  the  nodes  of 
the  American  economy  and  strike  the  enemy ’s  nodes. 


Osama  bin  Laden 


Background 
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•  In  early  FY07,  the  Electronic  Warfare  Directorate,  known  as  the 
412th  Electronic  Warfare  Group,  at  the  Air  Force  Flight  Test  Center 
(AFFTC),  Edwards  AFB,  commissioned  a  study  on  evolving 
threats. 

•  The  resulting  report,  “Survey  of  Evolving  Threats  &  Enduring 
Challenges”  was  delivered  in  October  2008. 

-  303  page  draft  report  delivered  -  still  needs  some  refinement  and  more 
information  added  on  certain  topics. 

-  Evolving  threats  covered  new  areas  out  to  10-15  years  and  enduring 
challenges  focused  on  existing  known  threats  and  their  continued  evolution. 

•  This  study  looked  at  both  U.S.  and  foreign  systems/threats. 

•  This  study  was  completed  by  Cubic  Applications  Inc.,  Threat 
Technologies  Division  in  conjunction  with  TYBRIN  Corporation. 


Background 
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•  The  focus  of  the  study  was  two  fold. 

•  It  looked  at  existing  and  new  evolving  threats  and  how  they  could 
potentially  impact  T&E  facilities  security. 

-  How  a  facility  is  planned  and  trained  to  confront  these  potential  threats 
(disaster  preparedness  plans,  etc.). 

-  External  and  potential  internal  threats  (insider). 

•  It  also  looked  at  how  these  evolving  threats  could  potentially 
impact  T&E  testing  (indoor,  outdoor)  and  T&E  infrastructure  in 
the  future. 

-  Related  to  the  type  of  T&E  testing  that  might  be  required  as  the  U.S. 
develops  new  weapon  systems  and  counter  measures  to  confront  these 
evolving  threats. 

-  Support  testing  related  to  the  Foreign  Material  Exploitation  Program 
(FMEP). 
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Evolving  Threats  Study 


•  The  study  looked  at  a  broad  spectrum  of  both  existing  threats  and 
new  evolving  threats. 

•  A  threat  description  was  provided  for  each  threat  examined. 

-  What  It  Is  and  the  Expected  Effects 

-  When  Available. 

-  Employment  Method. 

-  Ease  of  Employment. 

-  Credibility  of  Threat. 

-  Lethality  and  Range. 

-  Likely  Targets. 

-  Typical  Defense. 
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Evolving  Threats  Study 


•  The  threat  descriptions  provided  ratings  for  the  “Ease  of 
Employment”  and  “Credibility  of  Threat”. 

•  Ease  of  Employment  ratings. 

-  Difficult:  Little  or  no  potential  for  use  due  to  one  or  more  factors,  such  as 
lack  of  expertise,  materials,  or  deliver  system  being  readily  available. 

-  Moderate:  Possible,  but  one  or  more  factors  may  make  it  unlikely  at  the 
present. 

-  Unknown:  Ease  of  employment  unknown. 

-  Easy:  Relatively  easy  to  employ  due  to  readily  available  technology  and 
materials. 
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Evolving  Threats  Study 


•  Credibility  of  threat  ratings. 

-  Low:  Difficult  to  employ 

-  Moderate:  May  be  difficult  to  employ  at  present,  but  more  likely  as 
technology  develops  or  other  factors  come  into  play. 

-  Unknown:  Credibility  of  the  threat  is  unknown. 

-  High:  Likely  to  be  employed. 
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Evolving  Threats  Study 


•  The  study  also  provided  risk  factors  for  each  threat  examined. 

-  Potential  impact  on  DoD-cognizant  areas. 

-  Likelihood  to  be  used  against  critical  support  infrastructure. 

•  Near  term. 

•  Long  term. 

-  Assessed  Risk. 

•  Near  term. 

•  Long  term. 
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Evolving  Threats  Study 


•  The  risk  factors  identified  for  each  threat  provided  ratings  for  the 
“Likelihood  To  Be  Used”  and  “Assessed  Risk”. 

-  Likelihood  To  Be  Used  Against  Critical  Structures  (both  near  term  and  long 
term  risks). 

•  Low:  Little  or  no  potential  for  loss  of  a  given  critical  infrastructure 
segment,  denials  or  disruption  of  service  of  same. 

•  Moderate:  May  cause  loss  of  a  critical  infrastructure  segment,  or 
denial/significant  disruption  of  same. 


Unknown:  Unknown  impact. 

High:  Likely  to  cause  loss  of  one  or  more  critical  infrastructure 
segments,  significant  denial/disruption  of  service  of  same. 
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Evolving  Threats  Study 


•  Risk  factor  ratings  cont. . . 

-  Assessed  Risk  (both  near  term  and  long  term  risks). 

•  Low:  Little  or  no  potential  risk  for  the  time  period  indicated. 

•  Moderate:  Moderate  potential  risk  for  time  period  indicated. 

•  Unknown:  Unknown  risk. 

•  High:  High  potential  risk  for  time  period  indicated. 

•  For  each  threat  examined,  additional  information  was  provided. 

-  Threat  Environment:  Detailed  description  of  the  threat  itself  and  how,  when 
and  where  it  might  be  used. 

-  Key  Judgments:  Brief  description  of  the  threat’s  applicability  to  T&E. 

-  Observations:  Additional  information  if  necessary. 

-  Recommendations :  Information  on  how  to  address  each  threat,  types  of  testing 
that  may  be  necessary,  further  studies  needed,  etc. 
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Evolving  Threats  Study 

•  Threat  Description  Table  Example: 


Table  18-1.  Threat  Description:  Thermobaric  Materials 

What  It  Is 

Enhanced  blast  and  thermal  explosive  (similar  to  fuel-air  explosives);  bum  temperature  of  2,000°+C, 
but  blast  effects  are  more  serious  than  thermal. 

When  Available 

Now  From  Russia,  Bulgaria,  Poland,  Czech  Republic,  and  China. 

Employment  Method 

Same  as  any  other  explosive. 

Ease  of  Employment 

Easy  Easily  blended  and  transported;  delivered  like  any  other  munitions. 

Credibility  of  Threat 

High  Cheap,  simple  process  to  produce,  easy  to  employ. 

Lethality  and  Range 

More  energetic  and  far  greater  radii  of  effects  (impulse)  than  conventional  explosives;  flash  intensity 
and  duration  can  cause  temporary  blindness. 

Likely  Targets 

Especially  effective  within  closed  spaces,  e.g.  buildings  and  caves. 

Typical  Defense 

Barriers,  distance,  shock  absorption. 
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Evolving  Threats  Study 


•  Risk  Factors  Table  Example 

Table  18-2.  Risk  Factors:  Thermobaric  Materials 

Potential  Impact  on  DoD-Cognizant  Areas 

Personnel  affected,  but  in  relatively  small  area,  e.g.  buildings  and  blocks;  structures  and  materiel  destroyed  or  severely 
damaged. 


Likelihood  To  Be  Used  Against  Critical  Support  Infrastructure 


Near-Term: 

Moderate 

Simple  technology,  materials,  and  little  knowledge  needed. 

Long-Term: 

High 

High,  especially  if  terrorist  attacks  on  U.S.  soil  accelerate  after  U.S.  begins 
drawdown  of  U.S.  forces  from  Iraq. 

Assessed  Risk 


Near-Term:  Low 

Long-Term:  High 

•Thermobaric  materials  are  of  greatest  utility  in  destroying  structures  from  within.  Could  be  used  against  DoD  structures, 
personnel,  and  other  parts  of  the  national  security  infrastructure. 
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Evolving  Threats  Study 


•  Threats  examined  by  this  study  and  their  projected  impacts. 


Potential  Impact  On  Security 

Potential  Impact  on  T&E 
Infrastructure 

Potential  Impact  on  Both 
Security  &  T&E  Infrastructure 

No  Expected  Impact 

Biological  Agents 

Chemical  Warfare  Agents 

Toxic  Industrial  Chemicals  or 
Materials 

Interhalogen  Oxidizers  (IHO) 

Metal  Embrittlement 
Materials 

Improvised  Explosives 

Improvised  Explosive  Devices 
(IED) 

Vehicle-Borne  Improvised 
Explosive  Devices  (VBIED) 

Hatter  Charges 

Explosively  Formed 
Projectiles  (EFP) 

Energetic/Reactive  Materials 

Thermobaric  Materials 

High-Energy-Density 
Materials  (HEDM) 

Hafnium  Bombs  and  Isomer 
Grenades 

Information  Operations/Cyber 
Attacks 

Power  Disruption  Ordnance 

General  Nuclear  Devices  and 
Weapons 

Fourth  Generation  Nuclear 
Weapons 

Radiological  Dispersal 
Devices  (RDD)  and  "Dirty" 
Bombs 

General  Radio  Frequency 
Weapons  (RFW) 
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Evolving  Threats  Study 


•  Threats  examined  by  this  study  and  their  projected  impacts. 


Potential  Impact  On  Security 

Potential  Impact  on  T&E 
Infrastructure 

Potential  Impact  on  Both 
Security  &  T&E  Infrastructure 

No  Expected  Impact 

Charged  Aerosol 

Active  Denial  System  (ADS) 

Short-Pulse  Radars 

Foreign  RFW  Programs 

Hypersonic  Guns 

Electromagnetic  Rail  Guns 

Electrothermal  Chemical 
Guns  (ETC) 

Electric  Coil  Guns 

Acoustic  Guns 

Metal  Storm 

Laser  Weapons 

Dynamic  Pulse  Detonation 
(DPD)/Plasma  Acoustic 
Shield  System 

Particle  Beams 

Weather  Modifications 

Asymmetric  Warfare 

Unrestricted  Warefare 

Strategic  Indirect  Warfare 

Political  Warfare 

Economic  Warfare 

Anti-Satellite  Weapons 
(ASAT) 
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Appendices 

•  The  report  also  contains  eleven  different  appendices. 

-  Appendix  A:  Bibliography  -  A  detailed  10  page  list  of  all  the  sources  of 
information  used  to  compile  this  report. 

-  Appendix  B:  Glossary  -  A  list  of  all  the  unique  Terms/ Acronyms  used  in 
the  report  and  their  definition. 

-  Appendix  C:  Threat  Environments  &  Municipalities  -  This  appendix 
contains  5  pages  of  information  that  correlates  information  from  Table  C-l 
of  the  original  document  to  when  they  become  viable  and  whether  or  not 
they  may  effect  DoD  and  T&E  functions. 

-  Appendix  D:  Correlation  of  Adversary  Categories  of  Concern  Goals, 

Attack  Approaches,  Evolving  Threats,  and  DoD  T&E  Missions:  Five 

pages  of  documentation  regarding  the  categories  of  concern,  attack  goals, 
attack  approach,  and  weapons  of  choice  versus  DoD  T&E  missions. 
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Appendices 

•  The  report  also  contains  eleven  different  appendices  (cont. . .) 

-  Appendix  E:  Evolving  Threats  versus  Critical  Infrastructures  -  A  detailed  7 
page  report  that  lists  the  pros  and  cons  of  various  styles  of  attack  versus  the  targets 
of  choice,  delivery  means,  likelihood  of  target  choice  versus  effect,  and  most 
likely/least  likely  conflict  spectrum. 

-  Appendix  F:  Weapons  of  Information  Operations  &  Cyber  Warfare  -  A  three 
page  detailed  report  of  how  cyber  warfare  affects  the  U.S.  and  specifically  T&E 
ranges. 

-  Appendix  G:  Uranium  Enrichment,  critical  Mass,  and  Potential  Sources  of 

Special  Nuclear  Material  -  This  appendix  contains  6  pages  of  information  about 
the  percent  of  uranium  enrichment,  who  has  it  (especially  the  Highly  Enriched 
Uranium  (HEU)),  and  the  availability  of  this  material  for  weapon  programs 
underway  around  the  world. 

-  Appendix  H:  Foreign  RFW  Program  Summary:  This  appendix  contains  15 
pages  on  the  state  of  foreign  radio  frequency  weapons  programs  with  regards  to 
who  has  them,  how  far  along  they  are  in  the  development  and  deployment,  and  to 
what  extent  they  pose  a  threat  to  the  U.S. 
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Appendices 

•  The  report  also  contains  eleven  different  appendices  (cont. . .) 

-  Appendix  I;  Radio  Frequency  Weapon  Technology  -  This  appendix 
contains  13  pages  of  information  on  the  types  of  technology  being  looked  at 
by  the  various  nations  around  the  world  for  creating  radio  frequency  based 
weapons. 

-  Appendix  J:  Understanding  Radio  Frequency  Weapons  (RFWs)  -  This 
appendix  contains  16  pages  of  information  that  explain  why  RFWs  are  so 
important;  why  the  threat  is  so  great;  details  on  specific  RFWs  known  to 
exist;  how  they  interact  with  their  intended  target;  and  which  types  of 
targets  are  most  vulnerable  to  this  evolving  threat. 

-  Appendix  K:  Vulnerability  Mitigation  Program  Lessons  Learned  - 

This  appendix  contains  6  pages  of  information  on  nuclear  induced 
Electromagnetic  Pulse  (EMP)  effects;  current  protection  levels;  and  where 
we  need  to  focus  on  protection  for  future  generations  of  nuclear  induced 
EMP. 
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Summary 
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•  “Survey  of  Evolving  Threats  and  Enduring  challenges” 

-  Currently  in  a  draft  state. 

-  Final  report  due  out  later  this  year. 

-  Contains  classified  supplements. 

•  Comprehensive  report  on  existing  and  new  threats. 

•  Provides  detailed  information  on  each  threat  addressed. 

•  Provides  an  analysis  on  how  each  threat  potentially  impacts 
T&E. 

-  Facility  security  concerns. 

-  Potential  T&E  infrastructure  shortfalls. 
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Major  Conference  Themes/Issues 
wrt  Improving  T&E  Efficiency  and  M&S 

OT&E  should  be  validation  testing,  not  discovery  testing 

Mission  Based  Test  and  Evaluation 

Integrated  Testing 

Eliminate  failures  prior  to  test 

Unreliability  is  a  design  issue 

Put  the  “E”  back  into  T&E 

First  15%  of  investment  will  determine  or  dictate  the  remaining 
85%  of  LCC. 


Academy 

Associates 

Simplify,  Perfect,  Innovat 


■  Want  exit  criteria  from  one  phase  to  another 

■  Need  to  predict  performance  before  we  build  or  test 

©2009  Air  Academy  Associates,  LLC.  Do  Not  Reproduce.  Page  0 


Design  for  Breakthrough  Improvement 

(DFBI*) 


"Classic"  CPI/LSS 


focuses  here 


Research  Design  Development  Production 


Product  Stage 


AlR^ 

ACADEMY 

Associates 


•  Gain  knowledge  when  costs  are  lowest 

•  Design  in  quality  right  from  the  start 


*also  known  as  Design  for  Six  Sigma  (DFSS) 

©2009  Air  Academy  Associates,  LLC.  Do  Not  Reproduce.  Page  1 


Simplify,  Perfect,  Innovat 


DFBI  or  DFSS  Goals 


■  Reduce  Cycle  Time  in  the  Design  and 
Development  Process 

■  Reduce  the  total  resources  and  cost  over  the  life 
cycle 

■  Reduce  the  Cost  of  Poor  Quality 

■  Improve  Predictability  of  QCD  (Quality,  Cost, 
Delivery) 


AIR- 

At) 

ASSOCIATES 
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Simplify,  Perfect,  Innovat 


The  Benefits 


Air 


a* 


Ac  :ademy 
Associates 


Simplify,  Perfect,  Innovat 


Launch  Launch 

•  Early  problem  identification;  solution  when  costs  low 

•  Faster  market  entry:  earlier  revenue  stream,  longer 
patent  coverage 

•  Lower  total  development  cost 

•  Robust  product  at  market  entry:  delighted  customers 

•  Resources  available  for  next  game-changer 


Pre-DFSS: 
Reactive  Design 

•  Unhappy  customers  and 
employees 

•  Unplanned  resource  drain 

•  Skyrocketing  costs 

•  Next  product  compromised 


*  Upfront  investment  is  most  effective  and  efficient 

*  Show  customers  “high  quality”  products  right  from  the  start 

©2009  Air  Academy  Associates,  LLC.  Do  Not  Reproduce.  Page  3 
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6cm mx 
Associates 


The  Vision 


■  Reactive 
Design 


DFBI 


Predictive 

Design 


From 

To 

•  Evolving  design 

•  Disciplined  CTC 

requirements 

flowdown 

•  Extensive  design 

•  Controlled  design 

rework 

_ ^ 

parameters 

•  Product  performance 

m 

•  Product  performance 

assessed  by  “build 

r 

modeled  and  simulated 

and  test” 

•  Performance  and 

•  Desiqned  for  robust 

producibility 

performance  and 

problems  fixed  after 

producibility 

product  in  use 

•  Quality  “tested  in” 

•  Quality  “designed  in” 

•  Lean  Six  Sigma  (DMAIC)  fixes  known  problems. 

•  DFBI  or  DFSS  prevents  unknown  problems  from  occurring. 
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Simplify,  Perfect,  Innovat 


Background  -  MBT&E  Overview 

Mission-Based  Test  and  Evaluation 

is  a  methodology  that  focuses  T&E  on  the  mission 
task  capabilities  provided  to  the  warfighter.  It 
provides  a  framework  and  procedure  to: 

-  link  capabilities  to  the  attributes  of  the  materiel  system- 
of-systems; 

-  develop  evaluation  measures  that  assess  capabilities  and 
attributes; 

-and  link  the  evaluation  measures  to  all  available  data 
sources. 
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Linking  DFSS,  LSS,  and  DFR  in  the  QFD 

Flowdown 


Simplify,  Perfect,  Innovat 
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Transfer  Function:  The  Bridge  to 

Innovation 


Parameters 
or  Factors 
that 

Influence 
the  CTC* 


v 


y  =fi(x1,x2Ix3) 
s  =f2(x1;x2,  x3) 


Where  does  the  transfer  function  come  from? 

•  Exact  transfer  Function 

•  Approximations 

DOE  (also  known  as  Multi-Variate  Testing) 

Historical  Data  Analysis 

Simulation 


Simplify,  Perfect,  Innovat 


Critical  to  Customer  (or  Functional)  Performance  Measure 


©2009  Air  Academy  Associates,  LLC.  Do  Not  Reproduce. 
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Design  of  Experiments  (DOE)  and  Integrated  Testing 


“Integrated  testing  is  the 
collaborative  planning  and 
collaborative  execution  of  test 
phases  and  events  to  provide 
shared  data  in  support  of 
independent  analysis,  evaluation 
and  reporting  by  all  stakeholders, 
particularly  developmental  (both 
contractor  and  government)  and 
operational  test  and  evaluation 
communities.” 
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Simplify,  Perfect,  Innovat 


Design  of  Experiments  (DOE)  and  Integrated  Testing 


“Integrated  testing  is  the 
collaborative  planning  and 
collaborative  execution  of  test 
phases  and  events  to  provide 
shared  data  in  support  of 
independent  analysis,  evaluation 
and  reporting  by  all  stakeholders, 
particularly  developmental  (both 
contractor  and  government)  and 
operational  test  and  evaluation 
communities.” 
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Simplify,  Perfect,  Innovat 


What  Makes  DOE  so  Powerful? 
(Orthogonality:  both  vertical  and  horizontal  balance) 

A  Full  Factorial  Design  for  3  Factors  A,  B,  and  C,  Each  at  2  levels: 


Run 


A 


B 


AB 


AC 


BC  ABC 


+ 


\CAvmi 

Associates 


+ 
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Simplify,  Perfect,  Innovat 


Applications  of  Modeling  and  Simulation 


Power 


Automotive 


Aerospace 


Simulation  of  underhood  thermal  cooling  for  decrease 
in  engine  space  and  increase  in  cabin  space  and  comfort 


Simulation  of  stress  and  vibrations  of  turbine 
assembly  for  use  in  nuclear  power  generation 


Electronics 


Evaluation  of  dual  bird-strike  on  aircraft  engine 
nacelle  for  turbine  blade  containment  studies 


Evaluation  of  cooling  air  flow  behavior 
inside  a  computer  system  chassis 
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Simplify,  Perfect,  Innovat 


Examples  of  Computer  Aided  Engineering  (CAE)  and 

Simulation  Software 


Mechanical  motion:  Multibody  kinetics  and  dynamics 
ADAMS® 

DADS 


Implicit  Finite  Element  Analysis:  Linear  and  nonlinear 
statics,  dynamic  response 
MSC.Nastran™,  MSC.Marc™ 

ANSYS® 

Pro  MECHANICA 
ABAQUS®  Standard  and  Explicit 
ADINA 


\ 


AlR^ 

Academy 
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Simplify,  Perfect,  Innovat 


Explicit  Finite  Element  Analysis  :  Impact  simulation, 
metal  forming 
LS-DYNA 
RADIOSS 

PAM-CRASH®,  PAM-STAMP 

General  Computational  Fluid  Dynamics:  Internal  and 
external  flow  simulation 
STAR-CD 
CFX-4,  CFX-5 
FLUENT®,  FIDAP™ 

PowerFLOW® 
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Examples  of  High  Fidelity  Simulation  Models 


Preprocessing:  Finite  Element  Analysis  and 
Computational  Fluid  Dynamics  mesh  generation 
ICEM-CFD 
Gridgen 

Altair®  HyperMesh® 
l-deas® 

MSC.Patran 
TrueGrid® 

GridPro 
FEMB 
ANSA 


\ 


AlR^ 

Academy 

Associates 


Simplify,  Perfect,  Innovat 


Postprocessing:  Finite  Element  Analysis  and 
Computational  Fluid  Dynamics  results  visualization 
Altair®  HyperMesh® 
l-deas 
MSC.Patran 
FEMB 
EnSight 
FIELDVIEW 

ICEM  CFD  Visual3  2.0  (PVS) 

COVISE 
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Making  Use  of  Iterative  Simulation  and 
Modeling  for  Optimizing  the  Design 


©2009  Air  Academy  Associates,  LLC.  Do  Not  Reproduce. 
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Simplify,  Perfect,  Innovat 


Knowledge  Based  Principles* 


■  Develop  a  Product  Development  System  Built  Upon  Knowledge 

■  Focus  on  Knowledge  Generation  to  Streamline  the  Product 
Development  Process,  Rather  Than  Strict  Adherence  to  Pre¬ 
determined  Timelines 

■  Move  from  schedule-driven  acquisition  to  knowledge-driven 
acquisition 


Promote  Technical  Competency  Over  Procedural  Compliance 
Develop  Innovation-Driven  Project  Leaders/Program  Managers 


Air- 

At) 

ASSOCIATES 


■  Support  and  Reward  Technical  Excellence,  Systematic 
Innovation  and  Competency 

*  These  principles  are  based  on  the  key  findings  of  the  Research  and  Technology  Executive  Council, 
as  presented  in  its  September  2006  article  entitled  “Establishing  a  Lean  R&D  Organization 


©2009  Air  Academy  Associates,  LLC.  Do  Not  Reproduce. 
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Simplify,  Perfect,  Innovat 


Joint  Mission  Environment  Test 
Capability  (JMETC) 


Mr.  Bruce  Bailey 
Principal  Deputy  PM 

Joint  Mission  Environment  Test  Capability 
Test  Resource  Management  Center 

4  March,  2009 


The  JMETC  Mission 


JMETC  provides  the  DoD  T&E 
Community  the  resident  distributed 
test  expertise  and  the  persistent 
modern  network  infrastructure 

needed  for  the  connection  and  use  of 
distributed  live,  virtual,  and 
constructive  resources  to  augment 
the  DT&E  and  OT&E  of  joint  systems 
and  systems-of-system. 
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JMETC  Distributed  Test 
Infrastructure  Product  Lines 


•  Persistent  connectivity 

•  Middleware 

•  Standard  interface  definitions  and 
software  algorithms 

•  Distributed  test  support  tools 

•  Data  management  solutions 

•  Reuse  repository 
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L'<av 


JMETC  Enables 
Distributed  Testing 


Range 


Joint  Operational  Scenarios 


Hardware 
in  the 
Loop 
Lab 


TENA 

Standard 

Interface 

Definitions 


Installed 

Systems 

Test 

Facility 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


TENA 

Common 

Middleware 


TENA 

Common 

Middleware 


TENA 

Common 

Middleware 


TENA 

Common 

Middleware 


JMETC 
VPN  on 
SDREN 


JMETC 
Infrastructure 


Reuse 

Repository 


Distributed  Test 
Support  Tools 


Data  Management 
Solutions 


Systems 

Under 

Test 

Integrated 

Test 

Resources 


MS 


Environment 

Generator 


Threat 

Systems 


TENA 

Standard 

Interface 

Definitions 


TENA 

Standard 

Interface 

Definitions 


TENA 

Standard 

Interface 

Definitions 


Virtual 

Prototype 


TENA* 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Customer  Support 


*  TENA:  Test  and  Training  Enabling  Architecture 


JMETC... 

Reduces  Technical  Risk  2.  Saves  Time 

3.  Reduces  Costs 


•  People:  Experienced  and  highly  skilled  expertise  forward  deployed  for 
distributed  test  planning  and  operations 


•  Network:  Modern,  tested,  and  reliable  network  capability  already  in  place 


•  Practices:  Integration  and  interoperability  of  test  resources  is  proven 
practice  plus  a  quarterly  User’s  Group  to  capture  technical  requirements 


•  Data  Exchange:  Methodologies  and  solutions  have  already  been  tested, 
proven,  and  put  in  practice 


•  R&D  +  S&T :  JMETC  actively  captures  customer  needs  and  requirements  on  a 
continuous  basis  and  programs  basic  and  applied  research  projects  to 
improve  distributed  test  capabilities  for  the  Department 


•  Lessons  Learned:  JMETC  is  the  T&E  Communities’  enterprise-level  focal 
point  for  collecting  and  maintaining  “lessons  learned”  as  well  as 
implementing  resource  “reuse”  for  improving  the  DoD’s  distributed  test 
capability 

•  Tools:  JMETC  maintains  a  comprehensive  suite  of  distributed  test  tools... in 
fact,  the  same  tools  used  by  the  Joint  Interoperability  Test  Command  (JITC) 
for  net-ready  KPP  “interoperability”  certification” 
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JMETC  Leadership  &  Governance 

JMETC 


Chain  of  Command 
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Relationship  within  TRMC 
Synergy  through  Aligned  Investment 


Service 
T&E/S&T 
Working  Groups 


DoD  Strategic  Plan 
for  T&E  Resources 


TRMC 

Joint 

Investment 

Programs 


Quadrennial  Defense  Review 
Strategic  Planning  Guidance 
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i 
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Risk  mitigation  needs 
Technology  shortfalls 


Risk  mitigation  solutions 
Advanced  development 


Service 

T&E  Needs  and 
Solutions  Process 
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Requirements 

<: 


Capabilities 


Service  Improvement 

Acquisition  Programs  / 

T&E  Multi-Service  1 

DoD  Corporate 

&  Modernization/ 

Advanced  Concept 

/  Agency 

Distributed  Test 

Programs 

Technology  Demonstrations 

Capabilities 

Capability 

TRMC  Investment  Programs  Overview 


t&e/s&t  ifm 

CTEIP 

J  M  ETC 

•  Established  in  FY2002 

•  Develops  technologies 
required  to  test  future 
warfighting  capabilities 

•  6.3  RDT&E  funds 

•  ~$95M  /  year 

•  7  current  focus  areas 

•  Directed  Energy 

•  Hypersonics _ _ 

•  Established  in  FY1991 

•  Develops  or  improves 
test  capabilities  that 
have  multi-Service  utility 

•  6.4  RDT&E  funds 

•  ~$140M  /  year 

•  52  current  projects  ) 

•  27  projects  developing/ 
core  Joint  caoabUitteB 

-  2  projects  improving 
interoperability  test  cap. 

U  Established  in  FY2007\ 

•  Provides  corporate  ' 

infrastructure  for 
distributed  Joint  testing 

•  6.5  RDT&E  funds 

•  ~$10M  /  year 

•  33  current  sites 

•  Expanding  to  42  sites 

•  Maintains 

(^•Netcentric  Systems 

•  Network  connections 

•  Security  agreements 

•  Integration  software 

\  •  Interface  definitions  / 

1  •  Distributed  test  tools  / 

V  •  Reuse  repository  J  , 

•  Unmanned  Systems 

•  Multi-Spectral  Sensors 

•  Non-intrusive  Instruments 

•  Spectrum  Efficiencies 

•  9  projects  improyrng^-^ 
threat  representations 
used  in  testing 

•  16  projects  addressing 
near-term  OT  shortfalls 

' - - 

Ft  Lewis:  EPG 


Current  JMETC  Connectivity 

Dedicated,  trusted  VPN  on  SDREN  (part  of  the  GIG) 

Encrypted  for  Secret  -  System  High 

DISA-registered  IP  address  space 

Active  monitoring  of  network  performance 

Capable  of  supporting  multiple  simultaneous  test  events 


A  \  — O  Aberdeen:  ACCN 

SIMAFO  J  V'/N  Pax  River  (2):  ACETEF,  E2C 
Dahlgren  (2):  #f\-JMETC  SYSCON 
\  Boeing-St.  Louis:  J^CEDL,  IWSL  •TV  Langley:  GCIC 

Lokon:  HPCMO  JB,  JFCOM:  JSIC 

|  Dam  Neck:  CDSA 


China  Lake  (2):  AV-8B,  F/A-18 

#  \J  /  Kirtland  AFB: 

Pt  Mugu  (2):  Q  Edwards:  Ridley  O  SDOCC 

ITEC,  AEA  A  Camo  Pendleton:  MCTSSA 


Charleston  (2):  IPC,  MEF-MEU 


Pt  Loma:  ICSTD 

O  Army 
O  Air  Force 
O  Navy 
#  Marines 
O  Joint 
O  Industry 


WSMR:  IRCC 
Ft  H  uach  uCa:  Jfte^ 


Planned  JMETC  Connectivity 


Ft  Lewis:  EPG 


O  Army 
O  Air  Force 
O  Navy 
•  Marines 
O  Joint 
O  Industry 


Nellis:  CAOC-N 

\° 

China  Lake  (2):  AV-8B,  F/A-1 


Aberdeen:  ACCN 
ax  River  (2):  ACETEF,  E2C 
METC  SYSCON 
Langley:  GCIC 
JFCOM:  JSIC 
Dam  Neck:  CDSA 
Wallops:  NAVSEA 


(2>‘A' 

Pt  Mugu  (2): 
ITEC,  AEA 


Charleston  (2):  IPC,  MEF-MEU 


Pt  Loma  (2) 
ICSTD,  RLBTS 
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Network  Aggregation 
Bridging  Networks 
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Joint  Mission  Environment  Test 
Capability  (JMETC) 

FY08  Accomplishments 


JMETC  FY08  Customers 


•  Future  Combat  System  (FCS)  Combined 
Test  Office 

•  Joint  T&E  Methodology  (JTEM)  JT&E 

•  InterTEC 

•  Single  Integrated  Air  Picture  (SIAP)  JPO 

•  Air  Force  Datalink  Studies 


JMETC  FY08  Accomplishments 


•  Expanded  network  from  8  sites  to  32  sites 

•  Supported  multiple  tests  simultaneously 

•  Realized  reuse  from  persistent  agreements 

•  51  times  avoided  the  cost  of  re-doing  site 
connection  paperwork 

•  Characterized  JMETC  network  performance 
(latency,  throughput,  etc.)  in  detail 

•  Enhanced  the  JMETC  toolbox  with  test  tools 
for  OTH-G  and  USMTF  message  protocols 


Insight  Gained  by  the  DoD 


•  Timeliness  of  C2  processes 

•  Joint  Fires 

•  Joint  Close  Air  Support 

•  Combat  ID 

•  Airspace  deconfliction  studies 

•  Battlespace  management  between 
Army  and  Air  Force 

•  Link-16  implementation  differences 
between  Navy  and  Air  Force 

•  Improvements  to  Combat  Airborne 
Networking  gained  from  using 
FAST  over  Link-16 

•  Operational  Test  Assessment  of 
the  Capability  Test  Methodology 

•  Suitability  of  FCS  test  resources  to 
support  FCS  Milestone  C  decision 

•  Suitability  of  test  strategy  for  SIAP 
integration  software 


FY09  JMETC  Support  Schedule 


Customer 

Event 

Dates 

Air  Force 

Persistent  Fire  01 

October  08  -  December  09 

OSD/TRMC 

CTEIP  TENA  R6 

October  08  -  February  09 

Air  Force 

JEFX  09-02 

October  08  -  February  09 

Air  Force 

JEFX  09-03 

February  09  -  April  09 

Joint  Program 

Single  Integrated  Air 

Picture  (SIAP) 

October  08  -  June  09 

OSD/TRMC 

InterTEC  Tools 

Development 

FY09  -  Periodic 

Navy 

Joint  Surface  Warfare 

JCTD 

October  08  -  February  09 

Discussions  for  possible  future  teaming: 

Gerald  R.  Ford  Class  (CVN-21)  Joint  Strike  Fighter  (JSF) 

Multi-mission  Maritime  Aircraft  (MMA)  Net-Enabled  Command  Capability 

Future  Combat  System 
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Example: 

Single  Integrated  Air  Picture 
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Single  Integrated  Air  Picture  (SIAP) 


Real  time,  fire  control  quality 
Exploit  the  full  range  of  our  weapons 
Reduce  the  risk  of  fratricide  and  counter 
emerging  threats 


Vrt 


% 


°f/> 


*ti 


©/> 


2  Known  Friendly 
AWACS:  2  Known  Hostile 

1  Known  Cruise  Missile 


JLENS 


2  Known  Friendly 
E-2:  2  Known  Hostile 

1  Known  Cruise  Missile 


"Msst"  "Mss^ 


HOSTILE 


2  Known  Friendly 
AEGIS:  2  Known  Hostile 

1  Known  Cruise  Missile 


tIS 


CVN  (SSDS) 


USMC  Sensors 


*  ■ 


CAC2S: 


2  Known  Friendly 
2  Known  Hostile 
1  Known  Cruise  Missile 


Sentinel 


2  Known  Friendly 
IBCS:  2  Known  Hostile 

1  Known  Cruise  Missile 


JMETC  SIAP  Support 


Requirements 


r 

S 

IAP  JP< 

0 

k 

w 

Infrastructure 

Support 


Technical 

Support 


Middleware 


IMET( 

[ 

l  S 

Even! 

iuppo 

:  i 

rt  j 

Team 

'  1 

l"EN  / 
SDA 

V 
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JMETC  VPN  Sites 
Support  to  SIAP 

Joint  Combined  HWIL  Event  (JCHE)  5 


JMETC  SIAP  Support 


•  JMETC  provides  persistent  support  to  the  JPO  throughout 
entire  JCHE-5  schedule 


•  JMETC  distributed  and  Joint  testing  expertise  enables 
tailor-made  test  recommendations  to  SIAP 


•  Payoff  Example:  JMETC  involvement  enabled  early 
integration  testing  of  Wrap-Around  Simulation  Program 
(WASP)  Real-Time  Data  Distribution  System  (DDS) 

•  “Idea-to-Execution”  in  just  two  weeks 

•  Tactical  simulation  traffic  over  the  JMETC  VPN  five  months  before 
scheduled  distributed  site  integration  testing 

•  Issues  discovered  and  resolved  much  earlier  in  test  schedule 
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JMETC  Positive  Impact 
JEFX  09-01:  Persistent  Fire 


Global  Cyberspace  Integration  Center  Perspective 

JMETC-AF  partnership  accelerated  WWF  acquisition 
timeline 

•  Extended  limited  budget  resources  to  obtain  best  bang  for  the  buck: 
able  to  move  rapidly  from  concept  path  to  OT 

•  Reduced  certification  and  accreditation  issues  that  hamper 
developmental  activities 

•  Afforded  greater  opportunities  to  inject  direct  warfighter  input  without 
cost  of  reestablishing  network  infrastructure 


Greater  access  to  simulation  network  capabilities  increased  number  of 
test  cases  with  single  event 
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JMETC  Initiatives 


•  Streamlining  the  Network  Accreditation 
process 

*  Tiger  Team:  The  DoD  Information  Assurance 
Certification  and  Accreditation  Process  (DIACAP) 

•  Cross-Domain  Solutions 

*  Unified  Cross  Domain  Management  Office  (UCDMO) 

*  Information  Operations  Range 

*  Joint  Improvised  Explosive  Device  Defeat 
Organization 

•  West  Coast  Aggregation  Router 

•  Mobile  Node  Capability  (Transportable  Node) 
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Summary 


*  JMETC  supports  the  full  spectrum  of  Joint  testing, 
supporting  many  customers  in  many  different  Joint  mission 
threads 


*  JMETC  is  being  built  based  on  customer  requirements 

•  JMETC  event  support  can  be  tailored  to  customer  needs 

*  JMETC  is  partnering  with  Service  activities  and  leveraging 
existing  capabilities 

*  JMETC  is  coordinating  with  JFCOM  to  bridge  test  and 
training  capabilities 

*  JMETC  Users  Group  provides  an  open  forum  to  present 
emerging  requirements  as  well  as  new  technologies  & 
capabilities 

•  Next  JMETC  Users  Group  24-26  March,  San  Destin,  FL 


21 


JMETC  Program  Points  of  Contact 


JMETC  Program  Manager: 

Chip  Ferguson 

chip.ferauson@osd.mil 

703-604-0350  x138 

JMETC  Principal  Deputy  PM: 

Bruce  Bailey 

bruce.bailev(a)osd.mil 

703-604-0350  x144 

JMETC  Senior  Technical  Advisor: 

George  Rumford 

qeorqe.rumford@osd.mil 

703-601-5233 

JMETC  Lead  Systems  Engineer: 

Ryan  Norman 

rvan.norman@osd.mil 

703-604-0350x146 

JMETC  Website:  www.imetc.org 


Testing  &  Evaluation  for 
Weapon  System  Security 


March  3,  2009 

Office  of  the  Under  Secretary  of  Defense 
Acquisition,  Technology  and  Logistics 
Systems  and  Software  Engineering  Directorate 


Agenda 


*  Today’s  Threats,  Vulnerabilities 

*  Acquisition  Security  Policies 

*  Streamlining  Acquisition  Security 

-  Security  Disciplines 

-  Program  Protection  Plan 

-  Designing-In  Protection 

*  Implementing  Protection 

-  System  Component  Protection  Best  Practices  &  Tools 

-  Evaluation  of  Protection 

*  Summary 
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Today's  Throats  &  Vulnerabilities 


How  are  we  verifying  and  validating  that  we  are 
reducing  the  risk  of  these  types  of  attacks  in  our  systems? 


*  Threats:  Nation-state,  terrorist,  criminal,  rogue 
developer  who: 

-  Gain  control  of  IT/NSS/Weapons  through  supply  chain 
opportunities 

-  Exploit  vulnerabilities  remotely 

*  Vulnerabilities:  All  IT/NSS/Weapons  (incl.  systems, 
networks,  applications) 

-  Intentionally  implanted  logic  (e.g.,  back  doors,  logic  bombs, 
spyware) 

-  Commercial  software  and  circuit  cards  with  embedded  “phone 
home”  functionality 

*  Consequences:  Stolen  critical  data  &  technology; 
corruption,  denial  of  critical  warfighting  functionality3 


Improving  DoD  Program  Protection 
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Numerous  Security  Disciplines 


Program 

Protection 

(USD(I)) 


Anti- 

Tamper 


Configuration 

Manager 


X  Trusted  \ 
Micro¬ 
electronics 
DTICS  (DDRE 
\  Nil,  IP))  / 


System 

Assurance 

(AT&L/NII) 


Quality 

Engineer 


Software 

Protection 

Initiative 

(DDRE) 


Program 

Manager 


Trusted 

Foundry 

(DDRE) 


Center 

For 

Assured 

Software 

(NSA) 


Reliability  Y 
Engineer  /  Systems 

i  Engineer 


Safety 

Engineer 


Information 

Assurance 

(Nil) 


Protection  implemented  via  multiple 
initiatives  with  multiple  owners 


Acquisition  Security  Policies 


DODI  5200.39:  integration  point  for  policies,  NOT  replacement 


;  NEW  CPI  Protection 
■  Countermeasures 


Defense- 

In-Depth 

Intelligence 

Supply  Chain 
Engineering 
Certification 
Documented  Plan 


Critical 
Functionality 

Custom  COTS 


Component  Protection  Sought 

Critical  Critical 

Information  Technology 

JJ"-  ^  Software  Hardware/ 
Classified  Firmware 


Classified 


5000.1/2/SEP.  TEMP 


5200.39  CPI  Protection  &  PPF  5200.39  (legacy) 


CC/NIAP 


Supplier  Assurance  : 


IE 


FIPS 


.p.tLYaec...^ 


Engineerir 

g-ln-Depth 


1A 


■■CC/NIAP 


FIPS 


DIACAP 


Policy  Ownership 

USD(l) 

DoD  -  CIO/DSS 

DoD  -  AT&L 

DoD  -  AT&L/S&T  1 

DoD  -  CIO/DISA 

NSA/CC 

Dept,  of  State 

DoD  -  USD(l) 

NIST 

IA 


SPI 


OPSEC 


DIACAP 


Anti- 

Tamper 


5200.39  (legacy) 


DoD  Controlled 

Development/Operation 


Systems  Security  Engineer  Leads 
Integration  of  Security  Resources 


Missile  Technology 
Control  Regime 
(MTCR) 


Military  Critical 
Technologies  List 
{MCTL) 


Security 

Classification  Guide 
{SCG} 


Critical 
Technology 
^Resources  (CTR)^ 


New  technologies 
developed  in 
Defense  Science 
and  Technology 
Laboratory  {DSTL) 


C  Program  "'i 
\  KPPs  j 


^5-5230,28  j 


C 


Acquisition  Security  Database 


are  some 
of  the 
"  sources  " 
for 


Critical  Program 
Information  {CPI) 


T 


identifies 


Is  protected 


through 


c 


Is  protected  by 

♦ 


Countermeasures 


T 


D 


Include,  but  are  not  limited  to 

 T 


Program  Protection  Context 
Within  Systems  Engineering 


CPI  List  & 
Protection" 


Program  Protection  {PP)  is  a 
Risk  Mitigation,  System  Security 
Engineering  Process 
that  identifies  CP  I,  threats  and 
vulnerabilities,  and  selects  and 
implements  cost  effective 
countermeasures 


(r  draft) 


Technology 
Development 
Strategy  (TDS) 


PP  Status  I 


Acquisition  Strategy 


D 


PP  Detail 


^  ^Program  Protection^  Ian  {PPP}^ 


Documents 

Systems 

Security 

Engineering 

processes 
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Information  Assurance  Controls 

Supplier  Assurance  including  trusted  foundry 

Anti-T  amper 

Software  Protection  products 
Effective  software  development  processes 
Software  design  rigor 
Physical  security 
Export  Controls 
Foreign  Disclosure 


Documents 
plan  and  resources 
for  testing 
countermeasures 


& 


^  TEMP  J 


J 


New  PPP:  Data  Driven  Format 


Pithy,  Dynamic,  Modular 


Verbose,  Static,  Essay 


Critical  Program  Information  (< 

CPI) 

Critical  Program 
Information 

Impact  of 
Loss 

(Low,  Med, 

Hi) 

Reason  for  each  change  in  status) 

List  Locations 

(Lab(s),  PMO, 

Contractor  Name(s),  Test 
Site(s)) 

Status  Dates  (watch, 
new,  removed) 

GPS  j 

New:  Critical  warfighting  component 

PMO,  Contractor  X 

New  6/2006 

Radar  FPGA 

New:  target  for  hackers 

PMO,  Prime, 

Subcontractor  Z 

Watch  6/2007 

Communication 

Card 

rrtl^\ 

Watch:  US  lead  in  technology 

Removed:  No  longer  leading  edge 
technology 

N/A 

New  4/1998 

Removed  4/2007 

.  ^o'G 
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Early ,  Designed-In  Program  Protection 


•  Acquisition  Strategy,  TDS,  RFP,  SEP, 
and  TEMP  must  be  revised  to  include 
PPP  relevant  information 

•  Milestone  Decision  Authority 
approves  PPP  in  addition  to  PM 


•  Identify  draft  CPI,  estimated 
protection  duration  and  S&T  Lab 
countermeasures 


Streamlined  Program  Protection  Plan 

•  One-stop  shopping  for  documentation 
of  acquisition  program  security  (ISP, 
IAS,  AT  appendices) 

•  Living  document,  easy  to  update, 
maintain 

•  Improve  over  time  based  on  feedback 


MSA 


MS  B 


MS  C 


Full  Rate 
Prod  DR 


Materiel 

Solution 

Analysis 


/  .1 

Engineering  & 

Production  & 

TechDev 

CDD 

Manufacturing 

J  Development  & 
Demonstration 

CPD 

Deployment  O&S 

.  o 

•  Obtain  threat  assessments  from 
Intel/Cl,  assess  supplier  risks 

•  Develop  design  strategy  for  CPI 
protection 

•  Submit  PPP  to  Acquisition 
Security  Database  (ASDB) 


Contractor  adds  detail 
to  Program  Protection  Plan 
Preliminary  verification  and 
validation  that  design  meets 
assurance  plans 


Enhance  countermeasure 
information  in  Program  Protection 
Plan  (PPP) 

Evaluate  that  CPI  Protection  RFP 
requirements  have  been  met 
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Program  Protection  Toots 


CPI  Identification 


CPI  Protection  Measures 


Missile  Technology  Control 
Regime  (MTCR) 


Security  Classification 
Guide  (SCG) 


Military  Critical  Technologies 
List  (MCTL) 


Critical  Technology 
Resources  (CTR) 


Program  KPPs  &  WBS 


S-5230.28 


Technologies  developed  in 
Defense  Science  and 
Technology  Laboratory 
(DSTL) 


CPI  10  Tool 

Information, 

Technology, 

Functionality 


CPI  Risk 
Analysis 


Determine 
Consequence 
of  Loss 


Determine 

Exposure 

Level 


Systems 
Engineers 
Select  Level 
of  Protection 


{Requisition  Security  Data 

l^^TorTlbrizontal  Pro  fee 


Information 

Security 


IA 

Controls 


Supplier 

Assurance 

Best 

Practices 
Pick  List 
(new) 


CPI 

Protection 

Best 

Practices 
Pick  List 
(new) 


Best  Practices 
Tool 
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Best  Practice  Format 


DRAFT 

•  Title:  Name  of  best  practice 

•  Requirement:  Sample  requirement  language  for 
inclusion  in  RFP 

•  Application:  Explanation  of  conditions  under  which  best 
practice  should  be  applied 

•  Evaluation:  Recommended  technique  for  evaluation  for 
each  life  cycle  phase 

•  Metrics:  Criteria  for  successful  implementation 

•  Cost:  Rough  estimate  of  cost  (order  of  magnitude) 

•  References:  sources  of  information  and  SMEs  that 
contributed  to  development  of  this  control 

•  Background:  supporting  anecdotes/evidence  n 


Best  Practice  Example 


Not  Validated  For  Use  -  For  Example  Only 

•  Title:  Code  Static  Analysis 

•  Requirement:  Implement  static  code  analysis  tool  for 
use  during  software  development. 

•  Application:  automated  method  of  detecting  and 
eliminating  bugs  early  in  the  development  cycle 

•  Evaluation:  Analysis  of  code  improvements  and 
remaining  types  of  weaknesses 

•  Metric(s):  types  of  software  problems  eliminated 

•  Cost:  $250/user 

•  References:  samate.nist.gov,  DoD  Labs 

•  Background:  DoD  labs  and  commercial  vendors  have 
static  code  analysis  tools 
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CPI  Protection  Evaluation 


Forms  of  Evaluation: 


What  should  the  CPI  Verification 
and  Validation  Strategy  be  ? 


•  Analysis 

-  Pre-MS  B  analyze  planned  countermeasures  for  sufficiency 
versus  threats  and  vulnerabilities 


•  Testing 

-  System  Security  Certification? 

-  OT&E  attack  scenarios? 

-  DT&E  insider  attack  scenarios? 

-  Security  vulnerability  testing? 

-  Automated  identification  and  removal  of  malicious  code? 


•  Monitoring 

-  Survey  public  domain  information 

-  Detect,  record,  act  and  report  CPI  loss,  AT  breaches 
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Summary 


•  Program  Protection  strategy  provides 

-  Overarching  framework  and  process  to  integrate  acquisition 
security  policies  and  resources  early  in  the  life  cycle 

-  One-stop  shopping  for  acquisition  security  documentation 

-  Best  practice  tools  to  support  implementation 

*  Current  Test  and  Evaluation  resources  are  still 
fragmented  across  IA,  Anti-Tamper,  Software,  etc.  a 
comprehensive,  integrated  strategy  for  T&E  of 
program  protection  is  under  development 

We  welcome  feedback  on  PP  Streamlining  and  T&E 

www.aca.osd.mil/sse/ 

Christine.hines.ctr@osd.mil 
(703)  682-5309 
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QUESTIONS? 
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Streamlining  Program  Protection 


Improved 
Protection  of 
DoD  Weapon 
Systems 
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AIR  FORCE 


Test  and  Training  Enabling  Architecture  (TENA), 

An  Important  Component  in 
Joint  Mission  Environment  Test  Capability  (JMETC) 


Briefing  for: 

25th  Annual  NDIA  T&E  National  Conference 


March  4,  2009 

Gene  Hudgins,  TENA  SDA  User  Support  Lead 


What  is  JMETC? 


*  A  corporate  approach  for  linking  distributed  facilities 

•  Enables  customers  to  efficiently  evaluate  their  warfighting  capabilities 
in  a  Joint  context 

•  Provides  compatibility  between  test  and  training 

*  A  core,  reusable,  and  easily  reconfigurable  infrastructure 

•  Consists  of  the  following  products: 

•  Persistent  connectivity 

•  Middleware 

•  Standard  interface  definitions  and  software  algorithms 

•  Distributed  test  support  tools 

•  Data  management  solutions 

•  Reuse  repository 

*  Provides  customer  support  team  for  JMETC  products 
and  distributed  testing 


JMETC  Enables 
Distributed  Testing 


Joint  Operational  Scenarios 


Systems 

Under 

Test 

Integrated 

Test 

Resources 


y 


Virtual 

Prototype 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Hardware 
in  the 
Loop 
Lab 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Installed 

Systems 

Test 

Facility 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Range 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


JMETC 
Infrastructure 


Reuse 

Repository 


Distributed  Test 
Support  Tools 


Environment 

Generator 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Threat 

Systems 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Data  Management 
Solutions 


Customer  Support 


JMETC:  Here  and  Now 


*  Uses  the  Secure  Defense  Research  &  Engineering 
Network  (SDREN)  for  connectivity 

•  35  sites  currently  on-line 


*  Uses  Test  &  Training  Enabling  Architecture  (TENA) 

•  Gateways  to  link  to  existing  DIS  and  HLA  simulations 


*  Incorporates  InterTEC  test  tools 


*  Uses  the  JNTC-sponsored  Network  Aggregator  to  link 
together  other  networks 


•  Being  expanded  based  on  customer  requirements 


*  Holding  JMETC  Users  Group  meetings  to  discuss 
emerging  requirements  and  technical  solutions 

•  Seeking  the  “best  of  breed”  solutions  across  the  community 


ABOj 


JMETC:  Here  and  Now 


*  Uses  the  Secure  Defense  Research  &  Engineering 
Network  (SDREN)  for  connectivity 

•  35  sites  currently  on-line 

- \ 

*  Uses  Test  &  Training  Enabling  Architecture  (TENA) 

•  Gateways  to  link  to  existing  DIS  and  HLA  simulations 

_  _ / 

*  Incorporates  InterTEC  test  tools 


*  Uses  the  JNTC-sponsored  Network  Aggregator  to  link 
together  other  networks 


•  Being  expanded  based  on  customer  requirements 

*  Holding  JMETC  Users  Group  meetings  to  discuss 
emerging  requirements  and  technical  solutions 

•  Seeking  the  “best  of  breed”  solutions  across  the  community 


JMETC  Uses  TENA  to  Integrate  Sites 

(Can  gateway  to  existing  DIS  and  HLA  simulations 


•  TENA  is: 

•  Developed,  upgraded,  and  sustained  by  CTEIP  and  JNTC 

•  Middleware  that  provides  a  single,  universal  data  exchange  solution 

•  Common  for  test  and  for  training  (core  standard  in  JMETC  and  JNTC) 

•  Available  for  download  at  www.tena-sda.org  for  free 

•  TENA  provides: 

•  Interoperability  among  range  systems,  hardware-in-the-loop  laboratories, 
and  simulations  in  a  quick,  cost-efficient  manner 

•  A  capability  to  rapidly  and  reliably  develop  LVC  integrations 

•  A  set  of  community-agreed  object  models  that  define  the  data  elements 
used  in  LVC  integrations  -  maximizes  reuse  from  event  to  event 

•  An  auto-code  generator  to  drastically  reduce  TENA  incorporation  time 

•  Next  version  of  TENA  (version  6.0)  will: 

•  Provide  advanced  data  filtering  (only  data  of  interest  sent  over  the  wire) 

•  Improve  fault  tolerance  and  embedded  diagnostics 

•  Currently  being  beta-tested  for  a  formal  release  later  this  year 
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Gateway  Builder 


JMETC 


•  GWB  is  focused  on  integration  of  distributed  live,  virtual,  and  constructive 
(LVC)  systems  into  a  common  synthetic  battle  space  that  comprises  various 
simulation  protocols,  training  ranges,  live  systems  and  platforms 

•  Gateway  Builder  streamlines  integration  process  and  reduces  time  and 
effort  of  creating  gateways 

•  Gateway  Builder  is  a  flexible,  extensible,  graphically  driven  tool  that 
automatically 
generates  gateways 
to  bridge  simulation 
and  live  protocols 

•  Gateway  Builder 
supports  mappings 
between  TENA,  DIS, 
and  HLA  and 
message-based 
protocols  using 
any  object  model 


MSR  Gateway  Builder 


Select  Protocols 


'  ^  TEN  A 

-  Berne  nl 

-  Berne  nl 2 

-  Bemenl? 

-  Berne  nl-t 

-  Berne  nl£ 

-  Berne  nlG 


Map  Protocol  OMs 


TEN-, 

-  Be men  1 1 
Bemen  12 
Semen  13 
Bemen  I  + 
Bemen  15 
r-  Bemen  IG 


Generate 

Gateways 


12  Oct  2006 


Gateway  Builder  Simplified  Block  Diagram 
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TENA 


TENA  Overview 


•  Requirements 

•  Interoperability 

•  Reuse 

•  Composability 

•  Support  Rapid  Integration 

•  Gradual  Deployment 


•  Supports 

•  Testers  &  Trainers 

•  Joint,  Army,  Navy,  Air  Force,  Agencies 

•  Live,  Virtual,  Constructive 

•  Range,  Laboratories,  Simulations 

•  Real-Time  &  Non-Real-Time 

. n . TENA  Applications je^a  Toois''':::=:::: 


Guiding  Principles 

-  Provide  middleware 

-  Use  real  software  objects 

-  Maximize  code  generation 

-  Management  by  users  (AMT) 

-  No  license  fee  (GOTS) 


TENA  Utilities 


Non-TENA  Applications 


TEN  A  Architecture  Overview 


TENA  Applications 


TENA  Utilities 


Non-TENA  Applications 


Key  Release  6  Improvements 
and  New  Capabilities 


New  Middleware 

Metamodel  and  Model 

Capabilities 

Improvements 

■  Advanced  Filtering 

■  Fundamental  Sized  Type  Aliases 

■  OM  Subsetting  Support 

■  Const  Qualifier 

■  SDO  State  Processing  Support 

■  Optional  Attributes 

■  Self-Reflection  Option 

■  SDO  Initializers 

■  Object  Reactivation 

■  Middleware  Metadata 

■  Separate  Inbound/Outbound  ORBs 

■  Middleware  IDs 

New  Event  Management 

Usability 

Capabilities 

Improvements 

■  Object  Model  Consistency  Checking 

■  Observer  Pattern 

■  Remote  Object  Termination 

(with  Callback  Aggregation) 

■  Execution  Manager  Fault  Tolerance 

■  Local  Methods  Factory 

■  Embedded  Diagnostics 

Registration 

■  TENA  Console 

■  Code  Installation  Layout 
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Key  Release  6  Improvements 
and  New  Capabilities 


11 


Alaska  Training  Range  Evolution 
Program  (ATREP)  use  of  TENA 


ATREP’s  intent  is  to  enhance  the  existing  Pacific  Alaska  Range 
Complex  air  and  ground  capabilities  by  providing  a  force-on-force 
(FOF)  training  capability  that  fully  integrates  and  supports  joint  and 
coalition  components  for  both  air  and  ground  training  in  live,  virtual, 
and  constructive  (LVC)  domains. 


High  Side 
•TENA  ICADS 
•TENA  ACMI 
•TENA  9C2 
•TENA  DIADS 
•TENA  SimShield 


Low  Side 
•TENA  MOKKITS 
•TENA  MILES  2000 
•TENA  l-HITS 
•TENA  UMTE 


JMETC:  Here  and  Now 


*  Uses  the  Secure  Defense  Research  &  Engineering 
Network  (SDREN)  for  connectivity 

•  35  sites  currently  on-line 


*  Uses  Test  &  Training  Enabling  Architecture  (TENA) 

•  Gateways  to  link  to  existing  DIS  and  HLA  simulations 

/ - \ 

*  Incorporates  InterTEC  test  tools 

S _ J 

*  Uses  the  JNTC-sponsored  Network  Aggregator  to  link 
together  other  networks 

*  Being  expanded  based  on  customer  requirements 

*  Holding  JMETC  Users  Group  meetings  to  discuss 
emerging  requirements  and  technical  solutions 

•  Seeking  the  “best  of  breed”  solutions  across  the  community 


I OF 


InterTEC  Operational  View-1 

TENA-Based  Integrated  Test  Tool  Applications 


MM 


Rehearsal 

Control 

Monitoring 

Reporting 


Environment 


Data  Capture 
Stimulation 
Analysis 
Display 


Virtual  Components 
•HWIL 
Interfaces 
•  Message 
Generation 


Live  Components 


Constructive 

Components 

•  Simulation 
Interfaces 


Range 

Interfaces  ^ 
Range  ^  * ; 
Instrumentation 


r  .  .  20  Integrated  Apps  in  Spiral  2 

C4ISR  Instrumentation 

&  Analysis 


Joint  C4ISR  Test 


Test  Control 
•Planning 


InterTEC  Integration  with  JMETC 
Inextricably  Intertwined 


4Q  2010 

InterTEC 
Spiral  3 

4Q  2008 

InterTEC 
Spiral  2 

Fielded 

InterTEC 
Spiral  1 


IBS,  UAV, 
CGS  &  SOA 
Test  Tools 


OTH-G  & 
USMTF 
Test  Tools 


Link-16, 
Link-11, 
&  VMF 
Test  Tools 


JMETC 

Toolbox 


/ 

7 

JMETC 

Toolbox 

7 

/  / 

/ 

JMETC 
Toolbox  y 

/ 

JMETC  supports  InterTEC  during  their  spiral  development 

InterTEC  expands  JMETC  toolbox  with  certified  C4ISR  Test  Tools 

1  *  5 


rENA  Integrated  Development 
Environment  (TIDE) 


•  TIDE  is  a  tool  designed  to  assist  developers  in  the  creation, 
development,  testing  and  deployment  of  TENA  applications 

•  Initial  Capabilities 

•  Catalog  installed  object  models  on  a  user’s  machine 

•  Migrate  user  applications  between  object  model  versions 

•  Migrate  user  applications  between  middleware  versions 

•  Browse  and  download  object  models  available  in  the  TENA  Repository 

•  Request  object  model  distributions  from  the  TENA  Repository 

•  TIDE  2.0  is  the  current  version 

•  Available  at  http://www.tena-sda.org/tide  web  site 
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TENA  Tools  used  by  JMETC 
Interface  Verification  Tool  (IVT) 


•  Designed  to  support  the  integration  testing  of  TENA 
applications 

•  TENA  Standard  OM’s 

•  JNTC  and  InterTEC  LROM’s 

•  Provides  real-time  monitoring,  logging  and  statistics 
gathering 

•  Operates  in  three  different  roles,  either  stand-alone  or  in 
combination: 

•  Data  Subscriber  Role 

•  Data  Publisher  Role 

•  DIS  to  TENA  Gateway  Role 
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SIMDIS  UseofTENA 


SCORE  TSPI  Feed 


Southern 
California 


NRL 

Washington,  D( 


TENA 


•  Duration  testing  using  SCORE  TSPI 

•  Four  consecutive  days 

•  Win  XP,  Red  Hat  9,  Solaris  5.8 

•  Processed  180,000+  entities 


data 


feed 


•  Two  consecutive  days 

•  Win  XP,  Red  Hat  9 

•  Processed  53,000+  entities 

•  Results  and  observations 

•  No  issues  with  discovery  latency 

•  No  issues  with  update  latency 

•  No  issues  with  CPU  usage 

•  No  issues  with  memory  usage 


JMETC:  Here  and  Now 


*  Uses  the  Secure  Defense  Research  &  Engineering 
Network  (SDREN)  for  connectivity 

•  26  sites  currently  on-line 


*  Uses  Test  &  Training  Enabling  Architecture  (TENA) 

•  Gateways  to  link  to  existing  DIS  and  HLA  simulations 


*  Incorporates  InterTEC  test  tools 


*  Uses  the  JNTC-sponsored  Network  Aggregator  to  link 
together  other  networks 


•  Being  expanded  based  on  customer  requirements 

*  Holding  JMETC  Users  Group  meetings  to  discuss 
emerging  requirements  and  technical  solutions 

^  ♦  Seeking  the  “best  of  breed”  solutions  across  the  community  ^ 


JMETC  Users  Group  Meetings 


Identify  core  infrastructure  requirements  and  use  cases 

Discuss  available  solutions,  tools,  and  techniques 

Identify,  investigate,  &  resolve  issues 

Identify  opportunities  to  collaborate 

Share  lessons 
learned 


Users  Group  #01 

•  19-20  Jun  2007 

•  Dulles,  VA 

•  ~140  participai 

•  Plenary  sessk 

•  SIAP 

•  JSF 

•  FCS  CTO 

•  Tracks: 

•  User  Requir 

•  Security 

•  InterTEC  Spir^ 

•  Networking 


J5 


Users  Group  #07 

24-25  March 
AMT  41  on  26  March 
Ft  Walton  Beach,  FL 

Users  Group  #08 

23-24  June 
Portsmouth,  VA 


Users  Group  #03 

•  29-30  Jan  2008 
rtsmouth,  VA 
OlLPfiitici  pants 

briefs: 
Spiral  2 


Users  Group  #04 

•  20-21  May  2008 

•  Charleston,  SC 

•  ~135  participants 

•  Plenary  session: 

•  InterTEC  Spiral  2 

•  SPAWAR  Systems 
Center-Chariest 

•  Tracks: 

•  User 


Users  Group  #05 

•9-10  Sep  2008 

•  Boston,  MA 

•  ~176  participants 

•  Plenary  session: 

•  Dan  Roth,  AFFTC 

•  Rick  Cozby,  FCS 


equirements 
t  Tools 
d 
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Standard  Architecture  Management 

Team  Members 

(as  of  AMT-40) 


•  Air  Armament  Center  (AAC),  Eglin  AFB,  FL 

•  Naval  Undersea  Warfare  Center  (NUWC) 

•  Redstone  Technical  Test  Center  (RTTC) 

•  Electronic  Proving  Ground  (EPG) 

•  White  Sands  Missile  Range  (WSMR) 

•  Naval  Air  Warfare  Center  -  Aircraft  Division 

•  NAWC  -  Weapons  Division 

•  P5  Combat  Training  System  (P5CTS) 

•  Joint  National  Training  Capability  (JNTC) 

•  Pacific  Missile  Range  Facility  (PMRF) 

•  T&E/S&T  Non-lntrusive  Instrumentation 

•  Integrated  Network  Enhanced  Telemetry  (iNET) 

•  NAVSEA  Warfare  Center  -  Keyport 

•  Dugway  Proving  Ground  (DPG) 

•  Joint  Fires  Integration  &  Interoperability  Team  (JFIIT) 

•  Common  Range  Integration  Instrumentation  Systems  (CRIIS) 

•  Common  Training  Instrumentation  Architecture  (CTIA) 

•  Army  Operational  Test  Command  (OTC),  Fort  Hood,  TX 

•  Interoperability  Test  and  Evaluation  Capability  (InterTEC) 

•  Naval  Aviation  Training  Systems  Program  Office  (PMA-205) 

•  Air  Force  Flight  Test  Center  (AFFTC),  Edwards  AFB,  CA 

•  Aberdeen  Test  Center  (ATC),  Aberdeen  Proving  Ground,  MD 

•  Alaska  Training  Range  Evolution  Plan  (ATREP) 


Advising  AMT  Members 

(as  of  AMT-40) 


•  BMH  Associates,  Inc. 

•  Boeing 

•  Cubic  Defense 

•  DRS 

•  Embedded  Planet 

•  EMC 

•  MAK  Technologies 

•  NetAcquire 

•  Science  Applications  International  Corporation  (SAIC) 

•  Scientific  Research  Corporation  (SRC) 

•  Scientific  Solutions,  Inc.  (SSI) 


12  Oct  2006 
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Summary 


•  JMETC  supports  the  full  spectrum  of  Joint  testing, 
supporting  many  customers  in  many  different  Joint  mission 
threads 

•  CVN-21,  SIAP,  FCS,  JSF,  MM  A,  NECC,  DD1000,  WWF 

•  TENA  is  the  CTEIP  architecture  for  future  instrumentation, 
the  JNTC  architecture  for  Live  integration  and  an  enabling 
technology  for  JMETC 

•  TENA  and  JMETC: 

•  Being  built  based  on  customer  requirements 

•  Partnering  with  Service  activities  and  leveraging  existing 
capabilities 

•  Coordinating  with  JFCOM  to  bridge  test  and  training 
capabilities 

•  Provide  a  forum  for  users  to  develop  and  expand  the 
architecture 

•  JMETC  User  Groups,  TENA  AMT  Meetings 
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Important  Contact  Information 


•  TENA  Website:  www.tena-sda.org 

•  Download  TENA  Middleware 

•  JMETC  Website:  www.imetc.org 

•  TENA  Feedback:  feedback@tena-sda.org 

•  Provide  technical  feedback  on  TENA  Architecture  or  Middleware 

•  JMETC  Feedback:  imetc-feedback@imetc.org 

•  JMETC  Program  Office  Contact 

•  E-mail:  Telephone:  (703)  604-0350  ext.  0 
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3/13/2009  10:3( 


fBud  Irish  - 

SAIC  Vice  President  dMj 

Z-  •  Jk  jW " *  ' 

FCS  Integrated  Phases,  Simulatfbn 
&  Test  Deputy  IPT  MGR  •  ^ 


Approved  for  Public  Release,  Distribution  Unlimited,  PM  FCS  4  March  2009,  case  09-02' 


FUTURE  COMBAT  SYSTEMS 


Army  Leadership’s  View 


One  Tern-  TheAmy/Dsfensefitutehy 


“Future  Combat  Systems  is  the  core  of  our  modernization  effort  and  will 
provide  our  Soldiers  an  unparalleled  understanding  of  their  operational 
environment,  increased  precision  and  lethality,  and  enhanced 
survivability.” 


“We  believe  it’s  affordable  and  we  believe  it’s  an  investment  that  we  have 

make.  ffre  fyonora£,/e  pefe  Qeren  -  Secretary  of  the  Army 

Senate  Armed  Services  Committee 
February  26,  2008 


“We’re  listening  to  our  soldiers  and  commanders  in  the  field,  and  we  are 
giving  them  the  capabilities  they  need  -  as  fast  as  we  can  so  that  they  can 

Win  in  the  current  fight.  General  George  Casey,  Jr.  -  Chief  of  Staff,  U.S.  Army 

U.S.  Army  News  Release 
June  26,  2008 


The  Honorable  Pete  Geren 
Secretary  of  the  Army 


“Future  Combat  Systems  is  exactly  the  full-spectrum  system  that  we  need  Gen.  George  W.  Casey,  Jr. 

for  OUr  futUrG .  General  George  Casey,  Jr.  -  Chief  of  Staff,  U.S.  Army  Chief  of  Staff,  U.S.  Army 


Senate  Armed  Services  Committee 
February  26,  2008 


"Modernization  is  not  an  option.  “ 


"FCS  is  more  than  a  program,  it  is  an  Army  imperative. " 

The  Honorable  Dean  Popps  -  Acting  Assistant  Secretary  of 
the  Army  for  Acquisition,  Logistics  and  Technology 

February  2,  2008 


RH7 - 1 


The  Honorable  Dean  Popps 
Acting  ASAALT 


FCS  -  Army’s  #1  Modernization  Priority 


Approved  for  Public  Release,  Distribution  Unlimited,  PM  FCS  4  Mar  2009,  case  09-024. 
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Delivering  a  Versatile  21st  Century  Army 


FUTURE  COMBAT  SYSTEMS 


One  Tern- The  AtmyJDetenseftndusiry 


Non-Line  of 
Sight  Mortar 
(NLOS-M) 
XM1204 


Infantry  Combat 
Vehicle  (ICV) 
.XM1206  tJ 


Mounted  Combat 
System  (MCS) 
XM1202 


Armed  Robotic 
Vehicle  -  Assault 
Light  (ARV-A[L1) 
XM1219 

{ n  .Jfei 


Non-Line  of 
Sight  Cannon 
(NLOS-C) 
XM1203 


Network 


Non-Line  of  Sight 
Launch  System 
(NLOS-LS)  XM  501 


Medical  Vehicle 
Treatment 
(MV-T)  XM1208 


Class  I 

Unmanned  Air 
Vehicle  (UAV) 
XM  156 


Reconnaissance 
and  Surveillance 
Vehicle  (RSV)  l 
XM1201  A 


Medical  Vehicle 
Evacuation  (MV-E) 
XM1207 


Field  Recovery  and 
Maintenance  Vehicle 
(FRMV)  XM1205 


Command  and 
Control  Vehicle 
(C2V)  XM1209 


MULE-T 

XM1217 


MULE-C 

XM1218 


Class  IV 
Unmanned  Air 
Vehicle  (UAV) 
XM  157 


Tactical  and  Urban 
Unattended 
Ground  Sensors 


Small 

UGV 

(SUGV) 

XM1216 


Multifunctional  Utility/ 
Logistics  and  Equipment 
Countermine  and  Transport 


AN/GSR-10  AN/GSR-9 


Approved  for  public  release;  distribution  is  unlimit 


Recent  Program  Accomplishments 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TteAmy/Dsfens&fitutehy 


s  Completed  Non-Line-of-Sight  Cannon  PI  Prototype  delivery  and 
testing  to  support  NLOS-C  Milestone  C 


s  Completed  Spin  Out  1  Tactical  Field  Test,  Field  Demonstration, 
Test  and  Evaluation,  and  Preliminary  Limited  User  Test 


Executing  to  Support  Army  Modernization 


^  Completed  all  System  and  Platform  Preliminary  Design  Reviews, 
including  Class  I  and  IV  UAVs,  MULE  UGV,  Manned  Ground 
Vehicles  and  Network 


s  Completed  Integrated  Mission  Test  One 

^  System  of  Systems  Common  Operating  Environment  2.0 
Deliveries/Testing 

s  First  successful  Active  Protection  System  (APS)  End  to  End  Test 

s  Airborne  Standoff  Minefield  Detection  System  Captive  Flight 
Test 


Approved  for  public  release;  distribution  is  unlimited.  GOVT  24  Feb  2009,  Case  09-9028. 


Combined  Test  Organization  (CTO) 
Established  2004  -  Charter 


FUTURE  COMBAT  SYSTEMS 


One  TsamJbeAmyTDefmafiMtohy 


The  mission  of  the  CTO  is  to  ensure  the  planning  and  execution  of 
the  Future  Combat  System  (FCS)  Test  and  Evaluation  Program  at 
minimum  cost  and  duplication  of  effort  to  meet  developmental  and 

operational  testing  requirements. 


Approved  for  Public  Release,  Distribution  Unlimited,  PM  FCS  4  March  2009,  case  09-024. 
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Plan  Together,  Test  Once,  Share  the 
Data 


FUTURE  COMBAT  SYSTEMS 


■  Equal  Partnership  between  PMO,  LSI  and  ATEC 

•  Synchronizing  Developmental  and  Operational  test  planning 

•  Sharing  test  resources  and  support 

•  Collecting,  sharing  and  assessing  test  data  jointly 

•  Minimizing  duplication  of  test  support  and  time  required  to 
execute  combined  testing 

•  Preserving  OT  independence 

•  Reduce  Program  Risks  wherever  possible 


Approved  for  Public  Release,  Distribution  Unlimited,  PM  FCS  4  March  2009,  case  09-024. 
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CTO  Details  of  Operation 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  IheAtmyJDeteixsftndusiry 


■  Manage  all  FCS  T&E  funding  -  except  LSI  allocated 

•  Manage  all  FCS  Component,  System  Level,  and  System  of 
System  Level  developmental  and  integration  testing 

•  Approve  all  test  plans  for  Gov  and  LSI  testing 

•  Integrate,  coordinate,  and  plan  Combined  Developmental 
and  Operational  Testing  in  accordance  with  the  FCS  TEMP 

•  Support  Live  Fire  and  Operational  Testing 


Approved  for  Public  Release,  Distribution  Unlimited,  PM  FCS  4  March  2009,  case  09-024. 
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New  way  of  doing  business... 
brings  FCS  to  life . 


Doctrine 


Developing  New  Doctrine  and  Tactics , 
Techniques  &  Procedures  (TTPs) 


Personnel  & 
Facilities 


Executing  Network 

Enabled 

Operations 


Organization 


Training 


Leadership 


Materiel 


Real  Soldiers  and 
Leaders... 

Operating  in  live  and 
virtual  events... 

To  deliver  real  answers 


Conducting 
Integrated  Training 


Evaluating  Material 


Integrating  capabilities  for  the  current  and  future  modular  force!! 


As  of:  6  Feb  09 


Approved  for  public  release;  distribution  is  unlimited.  GOVT  24  Feb  2009,  Case  09-9028. 
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FUTURE  COMBAT  SYSTEMS 


Army  Evaluation  Task  Force  (AETF) 
In  Action 


One  Tern-  IheA/myJDeteixeftndusiry 


Approved  for  Public  Release,  Distribution  Unlimited,  PM  FCS  4  March  2009,  case  09-024. 


FUTURF  COMSAT  SYSTEMS 

AETF  P-LUT 

One  Teem-  TheAmy/Oefonsefindusify 


CL1  UAV  in  Overwatch 


UAS  Image  FBCB2 


SUGV  on  Point 


m  ;•#*  •*” 


■  ,  i 


_r«. 


.  i-, 

pr^l 


jr*. 


Emplacing  T-UGS 


.  r 


■ 


Emplacing  U-UGS 


Approved  for  Public  Release,  Distribution  Unlimited,  PM  FCS  4  March  2009,  case  09-024. 
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JEFX  08 — Mission  Accomplished 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  IheAmyJDeteiXGftndusiry 


Approved  for  Public  Release,  Distribution  Unlimited,  PM  FCS  4  March  2009,  case  09-024. 


Integrated  Mission  Test  -  1  (IMT1) 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  IheA/myJDeteixeftndusiry 


^Mission  Test  with  Soldiers 

-  45  Soldiers  used  a  prototype  Warfighter  Machine  Interface 
(WMI)  and  Battle  Command  System  in  a  virtual-constructive 
relevant  environment 

-  Soldier  feedback  on  WMI  design  and  cognitive  assessment  of 
the  WMI 

-  First  full  scale  system  of  systems  (SoS)  integration 

^System  of  System  Common  Operating  Environment 
(SOSCOE)  Scalability  and  Discovery  Test 

-  Large  scale  (100  platform)  network  emulation  at  the  service 
layer 

-  Identified  areas  to  reduce  network  load  and  improve 
robustness 

/ Common  Operating  Picture  (COP)  Dissemination  Test 

-  First  large  scale  (100  platform)  COP  emulation  at  the 
application  layer 

-  Confirmed  value  of  geographical  dissemination  on  local  COPs 
and  identified  areas  for  improvement 

^SoS  Simulation  Framework  Maturation  Test 

-  Confirmed  maturation  of  simulations  and  tools  to  support 
future  SoS  testing. 


Approved  for  Public  Release,  Distribution  Unlimited,  PM  FCS  4  March  2009,  case  09-024. 


FCS  Production  Activities 


FUTVRi  COMBAT  SYSTEMS 


One  Jem-  IheAmyJDetenseftndusiry 


Approved  for  public  release;  distribution  is  unlimited.  GOVT  24  Feb  2009,  Case  09-9028. 


BL. 


Summary 


FUTURE  COMBAT  SYSTEMS 


One  Tsam-lfmAmyTDefym/kxiusby 


■  Program  executing  to  achieve  successful  ‘09 
DAB 


•  Platform  and  Network  PDRs  complete; 
supporting  successful  Systems  of  Systems 
Preliminary  Design  Review 

•  FCS  providing  capabilities  to  current  force 

•  IBCT  TFT/FDT&E/LUT 

•  Supporting  testing  and  technologies  on  track 

•  FCS  Program  is  healthy  and  meeting 
commitments 


Delivering  a  Versatile  21st  Century  Army 


Approved  for  public  release;  distribution  is  unlimited.  GOVT  24  Feb  2009,  Case  09-9028. 


14 


N  A  I  R 

IJWWtk  I  AT  rC  I-  ■  l["iPUiai<t 

How  to  Incorporate  Modeling 
and  Simulation  into  the  System  Acquisition 

Lifecycle 

James  N.  Elele  PhD 


BATTIESPACE 

modeling  and  simulation 


NAVj^AI  r 


How  to  Incorporate  Modeling  and 

Simulation  into  the  System 
Acquisition  Lifecycle 


BATTIESPACI 


Presented  By 

Derek  Kropp 

25th  Annual  Test  &  Evaluation 
National  Conference  (March  2-5,  2009) 

National  Defense  Industry  Association  (NDIA) 

Atlantic  City,  NJ 

(For  Dr.  James  N.  Elele) 

NAVAIR 

Integrated  Battlespace  Simulation  &  Test  (IBST)  Dept.  (5.4) 

Battlespace  Modeling  &  Simulation  Division  (5.4.2) 

Battlespace  Modeling  &  Simulation  VV&A  Support  Branch 

Patuxent  River,  MD 

B700D0003 


NAVj^AI  r 

SUMMARY 


BATT1ESPACI 

fflfijUlinq  And  limuliliiiii 


•  DoD  &  Navy  Regulations  Mandating  the  Use  Of 
M&S  In  Acquisition  (DoD  5000.2-R) 

•  M&S  Support  in  the  System  Lifecycle  and  Systems 
Engineering  Technical  Review  Process 

•  M&S  Support  to  Phases  of  System  Lifecycle 

•  M&S  Support  to  the  System  Lifecycle 

-  Pre-system  acquisition:  Concept  Refinement 

-  Pre-system  acquisition:  Technology  Development 

-  System  acquisition:  Development/Demonstration 

-  System  acquisition:  Production  and  Deployment 


B700D0003 
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DoD  &  NAVY  REGULATIONS  REQUIRING  MODELING  & 
SIMULATION  USE  FOR  SYSTEM  LIFECYCLE  SUPPORT 


BATT1ESPACI 


Department  of  Defense 

DoD  Directive  5000.1,  “The  Defense 
Acquisition  System” 

DoD  Instruction  5000.2,  “Operation  of  the 
Defense  Acquisition  System” 

DoD  Directive  5000.59 

DoD  Acquisition  Model  and  Simulation 

Master  Plan 

DoD  Guide  to  System  of  Systems  (SoS) 
Engineering 

Army’s  Simulation  Based  Acquisition 


NAVAIR 


NAVAIRINST  4355.1 9C  -  NAVAIR 
System  Engineering  Technical 
Review  Requirements 
PEO  Aircraft  Carriers  Instruction 
5200.5 

NAVAIR  M&S  VV&A  Guidelines 
(2009  Draft) 


Department  of  the  Navy 

SECNAVINST  5000.2C  -  DON  Acquisition 
Policy 

SECNAVINST  5200.38A-  DoN  M&S 
Management 

OPNAVINST  5200.34  -  Navy  M&S 
Management 

M&S  VV&A  Implementation  Handbook, 
Volume  I  VV&A  Framework 
M&S  VV&A  Implementation  Handbook, 
Volume  II  VV&A:  Adopt,  Adapt  &  Improve 
(Draft) 


DoD  & 

Department  of 
the  Navy 
Simulation 
Support 
Documents 


0 


B 


B700D0003 
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DoD  REGULATION  5000.2-R 


BATTLESPACI 


•  Paragraph  3.4.4: 

-  Accredited  Models  and  Simulations  (M&S)  shall 
be  applied,  as  appropriate,  throughout  the 
system  lifecycle  in  support  of  various  acquisition 
activities. 

-  Note  that: 

•  M&S  shall  be  applied 

•  as  appropriate, 

•  throughout  the  system  lifecycle 


B700D0003 
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SYSTEM  LIFECYCLE  FRAMEWORK  OVERVIEW 


BATT1ESPACI 

m  u-d-iFEi  n  i)  And  liiriulilimi 


GOAL:  To  indicate  how  M&S  may  be  used  at  each  phase 


User  Needs  6t 

T ECHNOLOGY  OPPORTUNITIES 


B 


IOC 


FOC 


Concept 

Technology 

System  Development 

Production  & 

Operations  & 

Refinement 

Development 

&  Demonstration 

Deployment 

Support 

\  Concept, 
y  Dedifeon 

A 

f  )  Readiness 
Review 

a  rw> 

f  >  Dedston 

V  Review 
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M&S  in  the  System  Lifecycle  and  Systems  Engineering 
M&S  Support  In  The  Technical  Review  Process 


BATT1ESPACI 

m  u-d-nEi  li  ig  And  limilllljtl) 


AAA 


Concept  Refinement 

Technology 

Development 

System  System 

Integration  Demonstra¬ 
tion 

Critical 

Design 

Review 

System  Development 
&  Demonstration 

LRIP  Full-Rate 

Prod  & 
Deployment 

A  FRP 

\/  Decision 
Review 

Production  & 
Deployment 

V  Pre-Systems  Acquisition  J 

V  Systems  Acquisition  J 

^  Sustainment  j 

V  Y  Y 


Acquisition  Processes  and  M&S 

•  DoD  5000. 2-R,  Mandatory  Procedures  for  Major  Defense, 
Acquisition  Programs  (MDAPS)  and  Major  Automated 

Information  System  (MAIS) 

•  C3. 9.  MODELING  AND  SIMULATION  (M&S):  “The  PM 
shall  identify  and  fund  required  M&S  resources  early  in  the 
acquisition  lifecycle,  so  that  M&S  may  be  integrated  with  the 

How  to  Incorporate  M&S  into  System  Lifecycle 

•  Identify  applicable  existing  M&S 

•  Maximize  re-use  of  available  M&S 

•  Provide  product  growth  for  M&S  to  meet  intended  uses 

•  When  possible,  initiate  creation  of  collaborative  M&S 
environment  of  applicable  M&S 

T&E  program.” 

•  Paragraph  3.4.4 :  “PMs  shall  integrate  the  use  of  modeling 
and  simulation  within  program  planning  activities,  plan  for 
life-cycle  application,  support,  capitalizing  on  reuse  of 
models  and  simulations,  and  integrate  modeling  and 
simulation  across  the  functional  areas.” 

•  Provides  a  logical  start  to  the  program 

•  Maximizes  re-use  of  existing  M&S  providing  economical  testing 
methodology 

•  Presents  technology  opportunity 

What  is  the  Collaborative  M&S  Environment? 

•  Interactive  environment  populated  with  applicable  M&S  to  the 
specific  program 

•  Environment  provides  different  levels  of  fidelity  to  match 
different  phases  of  the  acquisition  process 

•  Ranges  from  desktop  engineering  models  to  HWIL  training 
simulators 

•  Provides  ready  availability  to  M&S  and/or  applicable  data  and 
information  to  system  designers,  developers,  testers,  operators 

B700D0003 
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M&S  SUPPORT  TO  THE  SYSTEM  LIFECYCLE 
PRE-SYSTEMS  ACQUISITION:  CONCEPT  REFINEMENT 


BATTIESPACI 


System  System 

LRIP  Full-Rate 

yzzzzzzzzgzzTm 

Integration  Demonstra¬ 

Prod  & 

tion 

Deployment 

Concept 

Refinement 

yv  Critical 

Desigi 

Review 

A  FRP 

Decision 

Review 

mgm 

Technology 

System  Development 

Production  & 

Development 

&  Demonstration 

Deployment 

Pre-Systems  Acquisition 

- Y - 


Systems  Acquisition 

Y 


,A 


Sustainment 

~ V — 


Lifecycle  Phase 

Pre-system  Acquisition: 

Concept  Refinement 
Technical  framework  definition 
Analysis  of  alternatives 
Capabilities  definition 
Rapid  virtual  prototyping 
Concept  exploration 
Augmentation  of  the  T&E  process 


SETR  Phase  Venues 

Initial  Technical  Review 
Alternative  System  Review 


M&S  Application 
Categories 

•  Architecture  Definition  •  Assembly  Simulations 

•  Effectiveness  Models  •  Campaign  Level 

•  Engineering  Models  Simulations 

•  Manufacturing  Simulations  •  Engagement  Level 

Simulations 

M&S  Used  For 

•  M&S  modification  •  Planned  capability 

requirements  assessments 

•  Concept  testing  •  Concept  development  input 

•  Gap  analysis  •  Technology  identification 

Example  M&S 
Products 

•  Draft  Initial  Capabilities  •  Plan  for  Analysis  of 

Document  Alternatives 

*  Initial  Capabilities  *  Planned  phased  funding 

Document  requirements  ; 

B700D0003 
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M&S  SUPPORT  TO  THE  SYSTEM  LIFECYCLE 
PRE-SYSTEMS  ACQUISITION:  TECHNOLOGY  DEVELOPMENT 


BATTLESPACI 

t*i«4*ILn.q  And  limulilJeq 


Concept 

Refinement 


Technology 

Development 


Pre-Systems  Acquisition 

- — , - 


Lifecycle  Phase 

Pre-system  Acquisition:  Technology 

Development 

Technological  risk  reduction 
Technology  integration  into  system 
Technology  analysis 
Capability  assessment 
Lifecycle  cost  projection 
Operational  test  integration  planning 
Virtual  prototyping 


SETR  Phase  Venues 
Systems  Readiness  Review 
Integrated  Baseline  Review 


System  System 
Integration  Demonstra¬ 
tion 


o 


Critical 

Design 

Review 


System  Development 
&  Demonstration 


LRIP 


o 


Full-Rate 
Prod  & 
Deployment 

FRP 

Decision 

Review 


Production  & 
Deployment 


j 


Systems  Acquisition 

- Y - 


Sustainment 

v 


M&S 

Application 

Categories 

•  Architecture  Definition 

•  Effectiveness  Models 

•  Design  and  Specialty 
Engineering  Models 

•  Manufacturing  Simulations 

•  Assembly  Simulations 

•  Supportability  Models 

M&S  Used  For 

*  Risk  reduction  assessment 

•  Technology  selection 

•  System  integration  analysis 

•  Interoperability  analysis 

Example  M&S 
Products 

•  Capabilities  Development 
Document 

•  Technology  Development 
Strategy 

•  Inputs  for  System  Readiness 
Review 

B700D0003 


8 


nav^Tai  r 

M&S  SUPPORT  TO  THE  SYSTEM  LIFECYCLE 
SYSTEMS  ACQUISITION:  SYSTEM  DEVELOPMENT/DEMONSTRATION  B  m5Z* * 

^  /  hi  u-d^Si  h  q  dud  liiriulilimi 


AAA 


System  System 
Integration  Demonstration 


System 

Development/ 

Demonstration 


Systems  Acquisition 

- Y - 


-A 


Sustainment  J 


Pre-Systems  Acquisition 


Lifecycle  Phase 
System  Acquisition 

Development/Demonstration 

•  Interface  requirements  definition 

•  Test  and  evaluation  of  technology 
under  development 

•  Hardware  prototype  T&E  process 
refinement 

•  Assessment  of  system  in  varying 
scenarios,  mission  space,  and 
performance  envelope 


•  Flight  Readiness  Review 

•  Operational  Test  Readiness  Review 

•  System  Verification  Review/Production 
Readiness  Review 


M&S  Application 
Categories 

•  System  Demonstration  •  Supportability  Models 

Models  •  HWIL/SWIL/MIL  T&E 

•  Effectiveness  Models 

M&S  Used  For 

•  System  integration  of  •  Analysis  for  refining  HW 

subsystems  and  components  and  SW 

•  Interoperability  analysis  •  System  integration  risk 

reduction 

Example  Products 

•  Prototypes  •  Capability  Production 

•  Risk  Mitigation  Reports  Document 

SETR  Phase  Venues 

•  Systems  Functional  Review 

•  Preliminary  Design  Review 

•  Critical  Design  Review 

•  Test  Readiness  Review 


B700D0003 
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M&S  SUPPORT  TO  THE  SYSTEM  LIFECYCLE 
SYSTEMS  ACQUISITION:  PRODUCTION  AND  DEPLOYMENT 


BATT1ESPACI 

mudp'iiiu  And  limuldliiiii 


A  ilk 


n  |  0 

Concept  Refinement 

KjgH 

_ 

System  System 

Integration  Demonstration 

t 

System  Development/ 
Demonstration 

LRIP  Full-Rate 

Prod  & 
Deployment 

Production 

and 

Deployment 

V  Pre-Systems  Acquisition  J 

Systems  Acquisition  Sustainment  J 

V -  V  Y 


Lifecycle  Phase 
System  Acquisition: 

Production  and  Deployment 

•  Detailed  system  design  development 

•  System  production  and  support 
process  definition 

•  Manufacturing  facility  design 

•  Production  flow  definition 

•  Production  bottleneck  analysis  and 
elimination 

•  Virtual  training 

SETR  Phase  Venues 

•  Physical  Configuration  Audit 


M&S  Application 
Categories 

•  Manufacturing  Simulations  •  Supportability  Models 

•  Assembly  Simulations  •  Training  Simulations 

M&S  Used  For 

*  Full  manufacturing *  *  Initial  production 

capability  analysis  recommendations 

•  Production  representative  •  Support  analysis 

statistics 

Example  M&S 
Products 

•  Full-rate  production  •  System  deployment  plans 

estimates  •  Operational  capability 

•  Process  Improvement  •  Quality  Control 

Report 

B700D0003 
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M&S  SUPPORT  TO  THE  SYSTEM  LIFECYCLE 
SUSTAINMENT:  OPERATIONS  AND  SUPPORT 


A 


A 


A 


jjljjp 

System  System 

LRIP  Full-Rate 

Sustainment  Disposal 

Integration  Demonstration 

Prod  & 

1^, 

System  Development/ 

Deployment 

Operations  & 

Technology 

Production  & 

Support 

Concept  Refinement 

Development 

Demonstration 

Deployment 

v: 


Pre-Systems  Acquisition 

- Y - 


"Tv 


Systems  Acquisition 

Y 


Sustainment 

Y - 
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Lifecycle  Phase 

Sustainment: 

Operation  and  Support 

•  Detailed  system  operational 
improvement 

•  Future  system  improvement 

and  support  and  process  definition 

•  Operational  bottleneck  analysis  and 
elimination 

•  Virtual  training 

•  Future  requirements  definition 

M&S  Application 
Categories 

•  Simulations  •  Supportability  Models 

•  Assembly  Simulations  •  Training  Simulations 

M&S  Used  For 

•  Full  manufacturing  •  Initial  production 

capability  analysis  recommendations 

•  Production  representative  •  Support  analysis 

statistics 

Example  Products 

•  Full-rate  production  •  System  deployment  plans 

estimates  •  Operational  capability 

SETR  Phase  Venues 

•  Physical  Configuration  Audit 

B700D0003 


11 


NAVj^AI  r 


REVIEW 


BATT1ESPACI 

mud-r'iiiu  And  limuljliiiii 


Acquisition  activities  span  all  phases  of  the  system  lifecycle 
M&S  heavily  supports  all  aspects  of  the  system  lifecycle 

M&S  provides  valuable  input  into  the  System  Engineering  Technical  Review  Process 


Requirements  Definition 


Program  Management 


Design  &  Engineering 


Test  Planning 


Result  Prediction 


Supplemental  T&E 


Manufacturing 


Logistics  Support 


Pre-Concept  Analysis 


Concept  Refinement 


JS 


& 


£ 


Technology  Development 


<>-► 


System  Development  and 
Demonstration 


o— ► 


Production  and 
Deployment 


Operations  and  Support 


M&S  Used  For 


Gap  and  Alternative 
Analysis 


Concept  Development, 
Technologies  Identification 


Theoretical  Integration  of 
Subsystems 


Production  Aid  by  Solving 
Problems  “Before”  They 
Occur 


Test  of  System  Upgrades 


B700D0003 
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Questions 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


25th  Annual  Test  &  Evaluation 
National  Conference 

Session  D 

Resources,  Instrumentation,  and 
Targets  Summary 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Brief  #1 
TENA 


•  Test  &  Training  Enabling  Architecture  (TENA) 
provides  a  standardized  software  interface  for 
interfacing  range  instrumentation  data  outputs. 

•  Version  6.0  is  schedule  for  release  soon. 

-  Many  new  improvements 

-  New  middleware  capability 

-  Metamodel  and  model  improvements 

-  New  event  management 

-  Usability  Improvements 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Brief  #1 

TENA  (cont...) 


•  InterTEC  -  New  CTEIP  program  to 
develop  more  tools  for  Joint  Mission 
Environment  Test  Capability  (JMETC). 

•  Interface  Verification  Tool 

-  Used  to  verify  and  validate  the  new  tools 
being  developed  for  use  with  TENA  and 
JMETC. 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Brief  #2 
JMETC 


•  Joint  Mission  Environment  Test  Capability 

•  New  means  for  linking  distributed  facilities 

-TENA  is  a  part  of  JMETC 

-Allow  joint  mission  environment  testing 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Brief  #3 

Evolving  Threats  -  How  They  Impact  T&E  Testing  & 

Infrastructure 

•  Outlined  a  study  conducted  by  AFFTC’s  412th 
Electronic  Warfare  Group. 

-  Focused  on  new  Evolving  Threats  and  existing  threats 
and  how  they  may  potentially  impact  T&E  infrastructure 
and  test  capabilities. 

-  Also  looked  at  the  potential  impact  to  T&E  facility 
security,  disaster  preparedness. 

•  Appeared  to  cover  quite  a  bit  of  what  DTRMC 
listed  as  focus  areas  for  their  strategic  plan  for 
T&E  MRTFB  ranges. 

-  Directed  Energy 

-  Chem/Bio 

-  Nuclear  EMP  Effects 

-  IED  and  counter  IED 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Brief  #4 


Threat/Target  Simulation  for  Live  Fire 


•  Provided  a  comprehensive  overview  of 
NAWCWD’s  threat  and  target  simulation 
capabilities 

-  Types  of  targets  available 

•  Land/Air/Sea 

-Types  of  threat  simulators  (systems)  available 

•  Land/Air/Sea 

-  Mockups  (visual  only) 

-  Mockups  (visual)  with  real  RF  emitters. 

-  RF  emitters  only 
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25th  Annual  Test  &  Evaluation 
National  Conference 

Session  VI 

T&E  Workforce  &  Resources 

Summary 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Brief  #1 


Today’s  and  Tomorrow’s  DoD  T&E  Acquisition  Workforce 


•  Observations: 

-  Fairly  wide  disparity  between  the  %  of  military  in 
the  T&E  workforce  between  the  Army,  Navy, 
and  Air  Force. 

-  A  significant  %  of  the  workforce  will  be  at  the 
retirement  age  in  the  next  10  years. 

•  Will  result  in  a  “brain  drain”  in  the  workforce. 

-  Current  economy  situation  may  lesson  the 
pending  “brain  drain”. 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Brief  #1 

Cont... 


•  Observations  (cont...) 

-  Data  used  indicates  the  workforce  has  remained 
relatively  stable  over  the  last  4  years. 

•  However,  the  contractor  workforce  was  not  included  in 
this  data. 

•  What  %  of  the  total  workforce  does  this  account  for? 

•  Lack  of  contractor  data  may  prove  this  to  be  inaccurate. 

•  Contractor  workforce  is  always  the  first  to  increase  or 
decrease  due  to  the  ebb  and  flow  in  funding  and 
workload. 

-  Changes  in  DoDI  5000.02  will  drive  changes  in 
the  T&E  workforce. 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Brief  #2 


The  DOD  Strategic  Plan  for  T&E  Resources 


•  Defense  Test  Resource  Management  Center  (DTRMC) 

-  Oversees  all  aspects  of  MRTFB  T&E  facilities  resources. 

•  Coordinates  operational  budgets 

•  Investments  (CTEIP  &  l&M)  for  facility  infrastructure. 

-  Responsible  for  generating  the  high  level  strategic  plan  for  T&E 
resources. 

-  Several  key  infrastructure  shortfall  focus  areas  appear  to  be  on 
“Evolving  Threats”  including: 

•  Directed  Energy  test  capability 

•  Chem/Bio  test  capability 

•  Nuclear  EMP  effects  testing  capability 

•  IED  and  Counter  IED  testing  capability 

-  Coordinating  with  each  Service  to  develop  a  viable  and  executable 
strategic  plan  to  ensure  maximum  utilization  of  available  resources. 


NDIA  T&E  National  Conference 


Session  B:  New  T&E  Policy  and  Policy 
Implementation  -  March  5,  2009 


Mr.  David  Duma 
Mr.  Robert  Wojciechowski 
Ms.  Darlene  Mosser-Kerner 
Mr.  Andy  Long 
Dr.  Beth  Wilson 
Ms.  Christine  Hines 
2nd  Lt.  Matthew  Steele 


Recommendations 


Use  new  TEMP,  cgjidance  and_  fomiat  (Reliability,  Integrated 
Testing,  and  Evaluation  Framework) 

Adopt  the  implementation  framework  strategy  for  Integrated 
Testing:  integrate  the  people,  the  planning,  and  the  data;  blend, 
harmonize 

Work  with  OUSD(AT&L)  to  develop  an  overarching  Critical 
Program  Ijifomiation  (CPI}  T&E  strategy. 

Incorporate  LFT&E  and_  survivability  as  part  of  evaluation 
framework  requirements  (implied  task) 

Include  contractors  in  the  T&E  process  (Title  X  does  not  prevent 
the  use  of  contractor  data  or  collaborative  planning  and  execution) 

Ensure  T&E  WIPTs  understand  their  duties  extend  beyond  writing 
the  TEMP;  responsible  for  T&E  execution  and  informing 
stakeholders 

Disseminate  new  policies  and_  definitions  (integrated  testing, 
mission-oriented  context) 


Issues/Potential  Future 
Conference  Topics  and  Sessions 

Biggest  barriers  to  integrated  testing  are  cultural 

Additional  changes  to  new  TEMP  guidance  and 
format  to  address  emerging  issues  (i.e.,  cyber, 
systems  assurance) 

Appropriate  level  of  detail  in  reliability  section  of 
the  TEMP 

Defining  and  justifying  contractor  participation 
early  (cost/benefit) 

Rapid  programs:  Schedule  vs.  rigorous  T&E 

Sharing  examples  and  lessons-learned  (include 
PMs) 


UA7i 


Secretary  Gates’  Recommendations  for 
Improving  Defense  Acquisition 
As  Submitted  Jan.  27,  2009 


Avoid  across-the-board  adjustments. 

Pursue  a  "75  percent  solution"  vs.  "exquisite"  systems. 

Invest  more  in  future-oriented  (joint)  programs. 

Freeze  requirements  at  contract  award,  write  contracts 
that  provide  incentive  for  “proper  behavior.” 

Seek  increased  competition  and  the  use  of  prototypes. 


Response  of  William  Lynn  to  Senate 
Armed  Services  Committee  Question  on 

Acquisition  Reform 

“Within  the  acquisition  process,  realism  and  stability  can 
be  fostered  through  greater  emphasis  on  independent 
assessments  of  costs,  technology  readiness,  and 
testing  maturity,  particularly  during  the  early  stages  of 
programs . ” 

-  Mr.  William  Lynn,  nominee  for  Deputy  Secretary  of 
defense  (Prepared  response  to  question  submitted  in 
advance  of  15  Jan  confirmation  hearing) 


“Facts  are  stubborn  things;  and  whatever 
may  be  our  wishes ,  our  inclinations,  or  the 
dictates  of  our  passion,  they  cannot  alter 
the  state  of  facts  and  evidence.  ” 


-  John  Adams 


New  Acquisition/T&E  Policies  in 

DoDI  5000.02 


The  DUSD(A&T)  shall  conduct  an 
independent  Assessment  of  Operational 
Test  Readiness  (AOTR). 

The  CAE  shall  consider  the  results  of  the 
AOTR. 


New  Acquisition/T&E  Policies  in 

DoDI  5000.02 


The  Technology  Development  Strategy 
shall  provide  for  two  or  more  prototypes. 


New  Acquisition/T&E  Policies  in 

DoDI  5000.02 


DT  and  OT  shall  be  integrated  and  seamless. 

Evaluations  shall  include  a  comparison  with 
current  mission  capabilities  .  .  take  into  account  all 
available  and  relevant  data. 

T&E  shall  be  conducted  in  a  continuum  of  live, 
virtual,  and  constructive  environments. 

T&E  should  assess  improvements  to  mission 
capability. 


New  Acquisition/T&E  Policies  in 

DoDI  5000.02 


PMs  for  all  programs  shall  formulate  a 
viable  RAM  strategy. 


Definition  of  Integrated  Testing  in 
Defense  Acquisition  Guidebook 

“Integrated  testing  is  the  collaborative  planning 
and  collaborative  execution  of  test  phases  and 
events  to  provide  shared  data  in  support  of 
independent  analysis,  evaluation  and  reporting  by 
all  stakeholders,  particularly  the  developmental 
(both  contractor  and  government)  and  operational 
test  and  evaluation  communities.  ” 
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Using  IMPRINT  to  Translate  Human  Performance  into  System  and 
Mission  Effectiveness 

Diane  Kuhl  Mitchell  and  Chameta  Samms 

March  3,  2009 


UNCLASSIFIED 


RDECOMJ) 


The  Challenge 


St 


System  and  Mission 
Effectiveness 


(Program 

Initiation} 


Human 
Performance 
Modeling 


*  *  ♦ 
•  * 

*  * 


Technology  Opportunities 
&  User  Needs 


Concept 
Refine  ment 


Technology 

Development 


<^>Concepl  .Decision 


System  Development 
&  Demonstration 

♦  Dr*l-i|h 

Readiness 

Rjcvicw 


-  Process  entry  at  or  C 

-  Entrance  criteria  met  before 
entering  plinse 

-  Evolutionary  Acquisition  or- Single 
Step  to  Full  Capability1 


LRIPrlOT-iE 


Production  & 
Deployment 

OFRP 

Review 


Pre-Systems  Acquisition 


Systems  Acquisition 


Operations  & 
Support 


Sustainment 
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rdeco/wj*  Why  Human  Performance  Modeling  (HPM)? 


Human  performance  is  challenging  to  predict 


Many  Variables 


Concept  System 


Too  Dangerous 


Field  Study  Not  Feasible 


System  Performance  =  f  ( human  performance) 
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INPUTS 


Gathered  from  such 

—  Time  and  accuracy  of  each  task  sources  as  existing 

—  Consequences  of  "poor"  performance  data,  algorithms,  and 

estimates  from  SMEs 


MODEL 


OUTPUTS 


Measures  of  effectiveness 


Not  descriptive  models,  but  predictive  models 

TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


Is  the  human  overloaded  with  tasks? 


Will  training  improve  human  and  system 
performance? 

f|haj|y  1  How  to  allocate  tasks  between 

automation? 


human(s)  and 


What  are  the  performance  tradeoffs  with  different  system 
designs  or  levels  of  operator  experience? 
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RDECOM 


Improved  Performance  Research 
Integration  Tool 


% 


IMPRINT  is... 

•  a  Human  System  Integration  tool 

•  a  dynamic,  stochastic  discrete  event  network  modeling  tool 


Improved  Performance  Research  Integration  Tool 


http://www.arl.arnriv.mil/IMPRINT 
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1970 


1980 


1990 


200 


Concept  Paper 
-  Air  Force  ~ 


MPT  data  provided 
-  Paper  &  pencil  - 


Automated  process 
-  Mini-computer  - 


MPT  link  to  performance 
-PC  - 


Integrated  analysis  environment 
-  Windows  - 


Goal  Oriented  Behaviors  &  HLA 
Compliance 


Navy  HARDMAN 

(Hardware  vs.  Manpower) 


*  Army  HARDMAN  II 


Army  HARDMAN 


IMPRINT  &  WinCrew 


New  output  reports  &  32  bit  application 


Tri-Service  analysis  capabilities,  New 
interface  &  improved  plug-in  and  links  to 
other  simulations 


IMPRINT  6 
IMPRINT  7 

IMPRINT  Pro 
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RDECOM 


What  can  you  do  with  IMPRINT? 


3 


•  Set  realistic  system 
requirements 

•  Identify  future  manpower 
&  personnel  constraints 

•  Evaluate  operator  &  crew 
workload 

•  Test  alternate  system- 
crew  function  allocations 

•  Assess  required 
maintenance  manhours 

•  Assess  performance 
during  extreme  conditions 


•  Examine  performance  as  a 
function  of  personnel 
characteristics  and  training 
frequency  &  recency 

•  Identify  areas  to  focus  test 
and  evaluation  resources 

•  Quantify  human  system 
integration  risks  in  mission 
performance  terms  to 
support  milestone  review 

•  Represent  humans  in 
federated  simulations 


IMPRINT  is  a  trade-off  analysis  tool 
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RDECOM 


Build  Operations  Model  with  IMPRINT 


Identify 

Mission 


Build 

Task 

Network 


Mission  Name 

Time  and  Accuracy  Standards 
Performance  Criteria 


Function  Names 
Task  Names 

Branching  Logic  (serial,  multiple,  prob.,  tactical) 


Time  &  Accuracy  Standards  &  Estimates 
Operator  Assignments 
Workload,  Taxons 


Characteristics,  Training,  &  Stressors 
Workload  Thresholds 


Run 
Model 


View 

Reports 


Time  Performance 
Accuracy  Performance 
Workload 


Mission  Performance 

-  Predicted  time  &  success  rate  of 
mission 

Function  Performance 

-  Predicted  time  &  success  rate  of 
individual  functions 


Reports 


Task  Performance 

-  Predicted  time  &  success  rate  of 
individual  tasks 

Operator  Workload 

-  Workload  over  time  for  each 
operator 

-  Tasks  performed  over  time  and 
associated  workload 
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RDECOM^  Human  System  Analysis  and  Testing  HB 


Army  Research  Laboratory 

Conduct  human  system  analysis  using  tools  such 
as  IMPRINT,  C3TRACE,  FAST,  JACK 


Program  Managers, 
Combat  Developers 

Redesign  systems  to  overcome  issues 
validated  during  test  &  evaluation 


Army  Test  and  Evaluation  Center, 

Army  Research  Laboratory 

Implement  issues  found  during  analysis  into  test  plans 
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RDECOfw  jP  Future  Tank  Manpower  Example  mh 


•  Rely  on  heavy  armor 
and  artillery  to  protect 
the  forces 

•  Heavy,  large  systems 
are  difficult  to  deploy 
rapidly 


•  Rely  on  situation 
awareness  to  protect 
the  forces 

•  Lighter,  smaller 
systems  are  easier  to 
deploy  rapidly 


Current  Tank 


Commander  Driver 


Gunner  Loader 


NEED  TO  REDUCE 
CREW  SIZE  TO 
SUPPORT  SMALLER, 
LIGHTER  VEHICLE 


Survivability  Onion 


Future  Tank 


Commander 


Driver 
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RDECOM 


Future  Tank  Manpower  Example 
Modeling  Approach 


fm 


•  Identified  functions  to  be 
completed  -  knowledge 
elicitation 

•  Set  up  experimental 
conditions  to  model 
based  on  varying  function 
allocations 

•  Built  models 

•  Validated  models  by 
walking-through  with 
Soldiers 

•  Completed  runs  and 
prepared  results 


Four  Conditions 

*  Gunner-Driver  and  Commander 

•  Commander-Driver  and  Gunner 

•  Commander-Gunner  and  Driver 

*  Commander,  Driver  and  Gunner 


Function 

Condition  1 

Condition  2 

Condition  3 

Condition  4 

Name 

GD  and  C 

CD  and  G 

CG  and  D 

C  and  G  and  D 

Function  allocation 

Function  allocation 

Function  allocation 

Function  allocation 

Drive 

GD 

CD 

D 

D 

Hindrance 

GD 

CD 

D 

D 

Remediate 

GD 

CD 

D 

D 

Engage 

GD  <C) 

0  (CD) 

CG 

G(C> 

Local  Security 

c 

G 

CG 

C  and  G 

External  Com 

c 

CD 

CG 

c 

Crew  Commo 

GD  &  C 

CD  &  G 

CG  &  D 

C&G&D 
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Number  of  Instances  in  Overload 


RnFCOMjk 


Future  Tank  Manpower  Example 

Results 


Worfl 


300 

250 

200 


O 


cc 

-te!  100— — 


100.00  200. ( 


Condition  1 

Gunner-Driver  and  Commander 


Commander  -  Driver  and  Gunner 

Highest  workload  of  all  conditions 


oad  for  Each  Condition 


Gunner  -  Driver  and  Commander 

No  shooting  on  the  move 


I 


Condition3  Condition  4 


Commander  -  Gunner  and  Driver 

Best  two  crewmember  function  allocation;  single  vehicle  commander 


■  Visual 
Auditory 


rgage 


GD 


tion 


Commander,  Driver  and  Gunner 

Two  crewmembers  scanning;  allows  hunter-killer  philosophy 


Condition  4 
C  and  G  and  D 


Function  allocation 


D 


C  and  G 


C&G&D 


Commander  Gunner-Driver 
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RDECOM 


Future  Tank  Manpower  Example 
Analysis  Impact 


2  Soldier  crew 
considered  HIGH  RISK 


•  Changed  the  crewmember  requirement  for  Operational 
and  Organizational  (0&0)  Concept  Document  and  the 
Operational  Requirements  Document  (ORD) 

•  Role  of  third  crewmember  changed  to  gunner  in  prime 
contractor  design  concept. 
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rdecomJ^ 

Future  Tank  is  part  of  Tank  Platoon  j 

3 


3 


3 


* 

t 

PL/VC 

02 

♦ 

Crew  Chief 

E5 

-  1  Q  LU 

4* 

ARV-Assault  (L) 
M240  7.62MM 

19A 

19K20 

19K10 

Javelin  (2) 

* 

t 

PSG/VC 

t 

Crew  Chief 

t 

DR 

E7 

19K40 

E5 

19K20 

E4 

19K10 

a 

CL  1  UAS  LCU  C 

* 

t 

VC 

E6 

19K30 

t 

Crew  Chief 
E5 

19K20 

t 

DR 

E4 

19K10 

XM36  120MM 
M2  50  CAL 
GMR 
WiN-T 
CC 


XM36  120MM 
M2  50  CAL 
GMR  8 
CC 


XM36  120MM 
MK19  40MM 
GMR  8 
CC 


Original  analysis 


Provides  BLOS  support  to  Infantry  Platoons 
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RDECOM 


Function-by-Function  Deconfliction 
Procedures  (North  &  Riley,  W/Index)* 


Response 

Visual 

Auditory 

Manual 

Verbal 

Visual 

HIGH  CONFLICT  (.7-.9) 
Directly  competing 
resources  (e.g.  two 
search  functions;  less  if 
functions  adjacent  or  on 
same  display  areas 

Auditory 

LOW  CONFLICT  (.2-.4) 
Noncompeting 
resources  (e.g.,  search 
and  listening). 

HIGH  CONFLICT  (.7-.9) 
Highly  competitive 
resources;  some  time¬ 
sharing  if  discriminability 
between  inputs  is  high 

Manual 

LOW  CONFLICT  (.1-.3) 

Noncompeting 

resources. 

LOW  CONFLICT  (.1-.3) 
Noncompeting  resources. 

HIGH  CONFLICT  (.7- 
.9)  Competing 
resources  such  as 
two  tracking 
functions  or  discrete 
choice  functions 
have  shown  high- 
dual  decrements. 

Verbal 

LOW  CONFLICT  (.1-.3) 

Noncompeting 

resources. 

MEDIUM  CONFLICT  (.4-.6) 
More  interference  if  task 
requires  voiced  output. 

LOW  CONFLICT  (.2- 
.4)  Noncompeting 
resources  (e.g., 
tracking  and  voice 
input). 

HIGH  CONFLICT  (1.0) 
Requires  complete 
serial  output;  e.g. 
giving  two  messages 
or  voice  commands. 

*  North,  R.  A.,  and  Riley,  V.  (1989).  "W/INDEX:  A  Predictive  Model  of  Operator  Workload."  Applications  of 

Human  Performance  Models  to  System  Design,  G.  McMillan  et  al.,  eds.  (Plenum  Press,  New  York,  NY).  ]A/!ARFIGHTER  FOCUSED 


Cognitive  Building  Blocks 


Continuous  Functions 

Discrete  Functions 

Vehicle 

Position 

Driving 

Monitor  Driver 

Local  Security 

Battle  Tracking 

Monitor  Vehicle  Intercom 

Monitor  UAV 

o 

3. 

O 

> 

73 

< 

Monitor  Platoon  TacNet 

Monitor  Company  TacNet 

Communicate  on  Vehicle 

Intercom 

Communicate  on  Platoon 

TacNet 

Communicate  on 

Company  TacNet 

LOS  Engagement 

Threat  Detected 

BDA 

Provide 

BLOS  Capability 

Intervene  ARV 

Intervene  with  UAV 

PL  MCS 

PL 

P 

P 

P 

P 

P 

r 

70% 

70% 

10% 

10% 

Crew  Chief 

P 

P 

70% 

10% 

100% 

Driver 

P 

PSG 

MCS 

PSGT 

P 

P 

P 

P 

70% 

70% 

10% 

10% 

Crew  Chief 

P 

P 

10% 

100%" 

Driver 

P 

MCS 

VC 

P 

P 

P 

70% 

100% 

100% 

Crew  Chief 

P 

100% 

Driver 

P 
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Performance  Decrement 


•  Crew  chief  (Gunner)  has  two  primary  functions 

•  Local  Security 

•  ARV  monitoring 

•  Both  visual  search  tasks 


•  90%  penalty  in  accuracy  on  one  of  the  two  concurrent 
functions 

•  If  local  security  than  9  out  10  targets  might  be  missed 

•  9  out  of  10  times  MCS  potentially  hit  and  destroyed 


•  HRED  experiment*  looking  at  concurrent 

performance  of  a  gunner’s  and  robotic  operator’s 
tasks  in  a  simulated  MCS  environment  supports  that 
local  security  will  be  the  function  degraded. 


*  Chen,  J.  Y.  C.,  &  Terrence,  P.  I.  (2007).  Effects  of  Tactile  Alerts  on  Concurrent  Performance  of  the 
Gunner's  and  Robotic  Operator’s  Tasks  in  a  Simulated  Mounted  Environment  (Tech.  Rep.  ARL-TR- 
4227).  APG,  MD:  U.S.  Army  Research  Laboratory. 
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RDECOM 


Impact  of  Predictions  on  Mission 

Performance 


mb 


Mounted  Supported  by  Dismount 

Urban/Mout 

Defensive  Operations 

Tactical  Move 

Attack 

Hasty  Defense 

Tactical  Move 

Attack 

Hasty  Defense 

Tactical  Move 

Defend 

Counterattack 

Tactical  Move 


I 


Platforms 
Library  of  blocks 
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Overall  Mission  Impacts 


•  Three  tanks  in  future  concept  platoon 

•  2  of  the  3  vehicles  have  gunners  monitoring  robotic  systems 

•  2  of  the  3  vehicles  have  gunners  potentially  missing  9  out  of  10 
threats 

•  10%  survivability 

•  Tank  platoon  mission  is  to  provide  fires  for  an  infantry 
platoon 

•  Infantry  platoon  has  reduced  protection 

•  All  vehicles  may  not  arrive  at  attack  start  point 

•  Company  mission  may  be  degraded 
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•  IMPRINT  Tool 

>  No  cost  to  government  employees  and  government  contractors. 

>  Email  IMPRINT-INFO@arl.army.mil 

•  Analytical  support 

>  Assistance  with  structuring  analysis. 

>  Analyses  completed  for  customers. 

>  Email  diane.k.mitchell@us.army.mil 
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TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


MANNED  FLIGHT  SIMULATOR  FACILITY  AT  PATUXENT  RIVER 


“Today’s  and 
Tomorrow’s  T&E 
Acquisition  Workforce” 

NDIA  -  25th  Annual  T&E 
National  Conference 
“New  Administration,  New 
Opportunities” 


March  4,  2009 

Ms.  Darlene  Mosser-Kerner 

Assistant  DD,  DT&E  Policy  and  Guidance 
Engineering  &  Test  Policy  &  Guidance 

OUSD(AT&L)/Systems  &  Software 
Engineering 


C-17  FLARE  EVALUATION  (GERMANY) 


“Today’s  and  Tomorrow’s  T&E  Acquisition  Workforce” 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  One  of  seven  mission  critical 
acquisition  career  fields 

•  Workforce  size,  composition, 
education,  retention,  recruitment, 
and  assignment  is  critical  to 
ensuring  accurate  and  complete 
support  to  acquisition  decision¬ 
makers 


•  Workforce  has  to  be  resourced  to 
meet  the  challenges  of  more 
complex  and  interdependent 
systems 

•  Potential  reduced  DoD  funding  will 
impact  the  T&E  support  efforts  and 
require  planning  for  and  obtaining 
multiple  use  T&E  data  early  and 
often  through  product/system 
development 


T&E  Community  must  be  flexible  and  adaptable 
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T&E  Mission  Challenge 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  319  Programs  on  the 
CY  09  OSD  T&E 


224  of  the  Programs  are 
under  DT&E  Oversight 


WHAT  DOES  TODAY’S  T&E  ACQUISITION 

WORKFORCE  LOOK  LIKE 


RETENTION 


CERTIFICATION 


T&E  IS  A  TEAM  PLAYER 


DEVELOPMENTAL  TEST  &  EVALUATION 


PROGRAM  SPONSOR  BUDGET  SCHEDULE 


T&E  FEELS  LIKE  EVERYONE  IS  AFTER  THEM  WHEN  THEY  HAVE  THE 

BALL 
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AT&L  Acquisition  Workforce 


Human  Capital  Fact  Sheet 


Defense  Acquisition  Workforce 

Acquisition,  Technology,  and  Logistics1 

Civilian  (Civ) 
AT&L 
Workforce 

Military  (Mil) 
AT&L 
Workforce 

Total 

AT&L 
Workforce 
(Civ  +  Mil) 

1  Size  &  Composition  || 

1  FY08  Workforce  Size 

110.934 

14.945 

125.879  | 

1  FY13  Planned/Budaeted  Size 

112.949 

15.899 

128.848  li 

B  Chanae  in  size  2005-2008 

-7% 

-3% 

-7%  1 

i  Civilian/Militarv  Comoosition 

88% 

12% 

1 

R  Educational  Attainment  1 

1  Bachelor’s  Dearee  or  Hiaher 

77% 

81% 

77%  | 

H  Graduate  Dearee 

26% 

43% 

28%  1 

N  Certification  (Cert)  SI 

H  Level  1  or  Hiaher 

73% 

61% 

72%  | 

i  Level  II  or  Hiaher 

63% 

41% 

61%  1 

1  Level  III 

38% 

19% 

36%  [| 

li  Position  Cert  Reauirement  Met 

62% 

45% 

60%  1 

11  Plannina  Considerations  HI 

%  Babv  Boomer/Traditional  Generations 

68% 

17% 

62% 

Averaae  Aae 

46.7 

36.3 

45.5 

- 

- 

Averaae  Years  of  Service 

17.8 

13.3 

17.3 

:K;I" 
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T&E  Workforce  as  part  of  the  AT&L  Workforce 


DEVELOPMENTAL  TEST  &  EVALUATION 


Human  Capital  Fact  Sheet 

Defense  Acquisition  Workforce 

Civilian  (Civ) 

Military  (Mil) 

Total 

Defense 

Test  and  Evaluation1 

T&E 

T&E 

T&E 

Acquisition 

Workforce 

Workforce 

Workforce 

Workforce 

(Civ  +  Mill 

Size  &  Composition 

FY08  Workforce  Size 

5.608 

1.812 

7.420 

125.879 

FY13  Planned/Budqeted  Size 

5.630 

1.834 

7.464 

127.539 

Chanae  in  size  2005-2008 

-4% 

14% 

0% 

-7% 

Civilian/Militarv  Composition 

76% 

24% 

88% /1 2% 

Educational  Attainment 

Bachelor’s  Dearee  or  Hiaher 

95% 

91% 

94% 

77% 

Graduate  Degree 

30% 

43% 

33% 

28% 

Certification  (Cert) _ 1 

Level  1  or  Hiaher 

76% 

49% 

69% 

72% 

Level  II  or  Higher 

68% 

24% 

57% 

61% 

Level  III 

52% 

8% 

41% 

36% 

Position  Cert  Requirement  Met 

67% 

29% 

58% 

60% 

1  Planning  Considerations 

%  Baby  Boomer/Traditional  Generations 

58% 

10% 

46% 

62% 

Average  Age 

43.8 

34.6 

41.5 

45.7 

Average  Years  of  Service 

15.4 

11.5 

14.5 

17.3 

AT&L  Headcount  by  Component 
(As  of  End  of  FY2008) 


DEVELOPMENTAL  TEST  &  EVALUATION 


Service/Agency 

C 

Acqu 

FY08 

ount  and  Co 
isition,  Techi 
FY08  (%) 

mposition  (F 
nology,  and  1 
Civ 

708) 

-ogistics 

Mil 

Civ  (%) 

Mil  (%) 

Army 

40,269 

32.0% 

38,689 

1,580 

96.1% 

3.9% 

Navy 

43,066 

34.2% 

38,579 

4,487 

89.6% 

10.4% 

Air  Force 

24,827 

19.7% 

16,067 

8,760 

64.7% 

35.3% 

DCMA 

7,334 

5.8% 

7,334 

0 

100.0% 

0.0% 

DLA 

3,927 

3.1% 

3,927 

0 

100.0% 

0.0% 

Other  Defense 

6,456 

5.1% 

6,338 

118 

98.2% 

1.8% 

Total 

125,879 

100.0% 

110,934 

14,945 

88.1% 

11.9% 
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T&E  Count  &  Composition 


DEVELOPMENTAL  TEST  &  EVALUATION 


Service/Agency 

C 

FY08 

ount  and  Co 

Test  and 
FY08 (%) 

imposition  (F 
1  Evaluation 
Civ 

755) 

Mil 

Civ  (%) 

Mil  (%) 

Army 

2,135 

28.8% 

2,117 

18 

99.2% 

0.8% 

Navy 

2,476 

33.4% 

2,034 

442 

82.1% 

17.9% 

Air  Force 

2,622 

35.3% 

1,276 

1,346 

48.7% 

51.3% 

DCMA 

31 

0.4% 

31 

0 

100.0% 

0.0% 

DLA 

1 

0.0% 

1 

0 

100.0% 

0.0% 

Other  Defense 

155 

2.1% 

149 

6 

96.1% 

3.9% 

Total 

7,420 

100.0% 

5,608 

1,812 

75.6% 

24.4% 

AT&L  Career  Field  Count  &  Composition 


DEVELOPMENTAL  TEST  &  EVALUATION 


Count  and  Composition 

Acquisition,  Technology,  and  Logistics 

Service/Agency 

FY08 

FY08  (%) 

Civ 

Mil 

Civ  (%) 

Mil  (%) 

Auditing 

3,638 

2.9% 

3,638 

0 

100.0% 

0.0% 

BCEFM 

7,085 

5.6% 

6,840 

245 

96.5% 

3.5% 

CON 

'  25,680 

20.4% 

21,773 

3,907 

84.8% 

15.2% 

FE 

|  4,920 

3.9% 

4,919 

1 

100.0% 

0.0% 

Property 

451 

0.4% 

451 

0 

100.0% 

0.0% 

IT 

i  3,934 

3.1% 

3,579 

355 

i  91.0% 

9.0% 

LCL 

13,361 

10.6% 

12,415 

946 

92.9% 

7.1% 

PQM 

9,138 

7.3% 

8,445 

693 

92.4% 

7.6% 

PM 

12,781 

10.2% 

8,070 

4,711 

63.1% 

36.9% 

Purchasing 

1,196 

1 .0% 

1,184 

12 

99.0% 

1 .0% 

SPRDE  (ST) 

480 

0.4% 

416 

64 

86.7% 

13.3% 

SPRDE  (SE) 

34,501 

27.4% 

32,385 

2,116 

j  93.9% 

6.1% 

SPRDE  (PSE) 

,  36 

0.0% 

36 

0 

100.0% 

0.0% 

T&E 

7,420 

5.9% 

5,608 

1,812 

75.6% 

24.4% 

Other 

129 

0.1% 

129 

0 

100.0% 

0.0% 

Not  Listed 

897 

0.7% 

817 

80 

91.1% 

8.9% 

Unknown 

232 

0.2% 

229 

3 

98.7% 

1 .3% 

Total 

125,879 

100.0% 

110,934 

14,945 

88.1% 

11.9% 
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AT &L  Acquisition  Workforce  Career  Fields 


DEVELOPMENTAL  TEST  &  EVALUATION 


Count  and  Composition 

Acquisition,  Technology,  and  Logistics 

Service/Agency 

2005 

2006 

2007 

2008 

2008  (%) 

Auditing 

3,536 

3,486 

2,852 

3,638 

2.9% 

BCEFM 

8,119 

7,747 

7,387 

7,085 

5.6% 

CON 

26,025 

27,748 

26,038 

25,680 

20.4% 

FE 

8,356 

3,927 

4,394 

4,920 

3.9% 

Property 

571 

530 

481 

451 

0.4% 

IT 

5,472 

4,843 

4,423 

3,934 

3.1% 

LCL 

12,493 

12,332 

12,604 

13,361 

10.6% 

PQM 

9,397 

8,966 

8,364 

9,138 

7.3% 

PM 

12,284 

12,775 

12,427 

12,781 

10.2% 

Purchasing 

2,438 

1,680 

1,170 

1,196 

1.0% 

SPRDE  (ST) 

314 

291 

483 

480 

0.4% 

SPRDE  (SE) 

34,752 

35,142 

34,710 

34,501 

27.4% 

SPRDE  (PSE) 

0 

0 

0 

36 

0.0% 

T&E 

7,384 

7,280 

7,419 

7,420 

5.9% 

Other 

237 

465 

98 

129 

0.1% 

Not  Listed 

320 

811 

3,070 

897 

0.7% 

Unknown 

2,672 

219 

112 

232 

0.2% 

Total 

134,370 

128,242 

126,032 

125,879 

100.0% 
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T&E  Occupational  Codes 


DEVELOPMENTAL  TEST  &  EVALUATION 


Top  5  OC 
Test  and  Evalu 

OCC  Series  ■  Description 

C  Series 

ation  (Civilian) 
Total 

Total  (%) 

Cumulative  (%) 

0855  -  Engineer,  Electronics 

1,644 

22.2% 

22.2% 

0801  -  Engineer,  General 

1,110 

15.0% 

37.1% 

0830  -  Engineer,  Mechanical 

662 

8.9% 

46.0% 

0861  -  Engineer,  Aerospace 

394 

5.3% 

51.3% 

1515  -  Operations  Research  Analyst 

389 

5.2% 

56.6% 

#0cc  Series  in  Career  Field  =  55 
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AT&L  T&E  Headcount  by  Component 

(As  of  End  of  FY2008) 


DEVELOPMENTAL  TEST  &  EVALUATION 


Count  and  Composition  (FY08) 

Test  and  Evaluation 

Service/Agency 

Count 

Army 

2135 

Navy 

2360 

Marine  Corps 

116 

Air  Force 

2622 

Defense  Contract  Management  Agency  (DD63) 

31 

Defense  Information  Systems  Agency  (DD04) 

37 

Defense  Logistics  Agency  (DD07) 

1 

Defense  Threat  Reduction  Agency  (DD61) 

11 

Dept  of  Defense  Test  Resource  Mgmt  Center  (DD68) 

6 

Missle  Defense  Agency  (DD27) 

86 

National  Defense  University  (DD69) 

1 

Office  of  Inspector  General  (DD26) 

1 

Organization  of  the  Joint  Chiefs  of  Staff  (DD02)/Defense  System  Management  College 

6 

Under-Secretary  of  Defense/AT&L  (OUSD/AT&L) 

7 

Total 

7420 
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8,000 

7,000 

i?  non 

b,000 

c  non 

5,000 

>i  Ann 

4,000 

o  nnn 

3,000 

2,000 

a  nnn 

1,000 

0 

2005 

2006 

2007 

2008 

nOther  Defense 

i  49 

86 

97 

155 

□  DLA 

2 

2 

10 

1 

□DCMA 

3 

6 

36 

31 

e  Air  Force 

2,416 

2,598 

2,592 

2,622 

■  Navy 

2,453 

2,447 

2,549 

2,476 

■  Army 

2,461 

2,141 

2,135 

2,135 

Total  7,384  7,280  7,419  7,420 
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T&E  acquisition  work  force  composition  by 

generations 


DEVELOPMENTAL  TEST  &  EVALUATION 


Generation 

National  Workforce* 

DoD  Workforce 
(Civilian)** 

Defense  Acquisition 
Workforce  (Civilian)*** 

Defense  Acquisition 

T&E  Workforce 
(Civilian)**** 

Workforce 

(millions) 

% 

Workforce 

Workforce 

% 

Workforce 

Workforce 

% 

Workforce 

Workforce 

% 

Workforce 

Traditional 
(born  before  1946) 

7.4 

4.8% 

45,625 

6.7% 

6,624 

5.9% 

237 

4.2% 

Baby  Boomers 
(1946-64) 

56.7 

36.5% 

438,971 

64.5% 

74,887 

67.3% 

3,327 

59.3% 

Generation  X 
(1965-76) 

41.8 

26.9% 

132,948 

19.5% 

18,544 

16.7% 

1,127 

20.1% 

Generation  Y 
(1977-1989) 

42.8 

27.5% 

62,676 

9.2% 

11,286 

10.1% 

921 

16.4% 

Millenium 

(1990-present) 

6.6 

4.3% 

153 

0.0% 

0 

0.0% 

0 

0% 

Notes: 


*Source:  Projected  2010  workforce  size  -  DAU  analysis  using  Bureau  of  Labor  Statistics  workforce  participation  data  (Dec  2007)  &  US  Census  Bureau  population  data  (2004)  (best 
available  data  from  agency  web  sites  as  of  July  2008) 

**Source:  OSD  P&R  Report:  DoD  Civilian  Workforce  Statistics/DoD  Demographics/May2006  Edition 
***Source:  AT&L  Datamart  FY07  AT&L  Workforce  Count/AT&L  workforce  data;  does  not  contain  456  files  with  null  for  age 
****Source:  AT&L  Datamart  FY07  AT&L  Workforce  Count/BCEFM  workforce  data;  does  not  contain  12  files  with  null  for  age 
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Career  Field 


Certification  Rates  by  Career  Field 

AT&L  (FY08) 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 


Auditing 

BCEFM 

CON 

FE 

Property 

IT 

LCL 

PQM 

PM 

Purchasing 
SPRDE  (ST) 
SPRDE  (SE) 
SPRDE  (PSE) 
T&E 
Other 
Not  Listed 
Unknown 
Total 


I 


■  Meets  or  exceeds  position  certification  requirements 
Position  certification  requirements  Not  Met* 

■  Unknown 


Career  Field 


o% 


10% 


20% 


Certification  Rates  by  Career  Field 

T&E  (FY08) 

40%  50% 


30% 


60% 


70% 


80% 


90% 


100% 


Army 


Navy 


Air  Force 


DCMA 


DLA 


Other  Defense 


Total 


62.5% 

37.5% 

■  ■ 

67.1% 

32.9% 

44.4% 

55.6% 

35.5% 

64.5% 

1  100.0% 

66.4% 

26.3% 

7.2% 

57.6% 

42.2% 

0.' 

I  Meets  or  exceeds  position  certification  requirements 
Position  certification  requirements  Not  Met* 

IUnknown 

‘The  “Not  Met”  category  includes  workforce  members  who  have  been  in  their  positions  for  less  than  24  months. 
These  w  orkforce  members  are  in  a  “Grace  Period”  and  in  compliance  w  ith  DAWIA  policy 
Source:  AT&L  Workforce  Datamart 


WHAT  WILL  TOMORROW’S  T&E  ACQUISITION 

WORKFORCE  LOOK  LIKE 


RETENTION 


CERTIFICATION 


Tomorrow’s  T&E  Challenges 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  DoDI  5000.02  Changes  affecting  T&E 
mission: 

>  Materiel  Development  Decision  Review 
> Competitive  Prototyping 
>More  Frequent  Program  Reviews 
> Configuration  Steering  Boards 
> Integrated  DT  and  OT 
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SEC.  852.  DoD  ACQUISITION  WORK  FORCE 
DEVELOPMENT  FUND  NDAA  FY  2008 


DEVELOPMENTAL  TEST  &  EVALUATION 


”...(a)  Establishment-  The  Secretary  of  Defense 
shall  establish  a  fund  to  be  known  as  the 
'Department  of  Defense  Acquisition  Workforce 
Fund'  (in  this  section  referred  to  as  the  'Fund')  to 
provide  funds,  in  addition  to  other  funds  that 
may  be  available,  for  the  recruitment,  training, 
and  retention  of  acquisition  personnel  of  the 
Department  of  Defense.” 
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SEC.  852.  DoD  ACQUISITION  WORK  FORCE 
DEVELOPMENT  FUND  NDAA  FY  2008 

DEVELOPMENTAL  TEST  &  EVALUATION 

•  Eleven  Initiative  areas  to  focus  fund  allocation  for  training, 
retention,  and  recruitment 


•  Components  and  Agencies  to  identify  specific  items  for  each 
Initiative  for  its  acquisition  career  fields 


1 

Training  Enhancement  &  Capacity  Expansion 

2 

Comprehensive  Acquisition  Workforce  and  Student  1  nformation  System 

3 

Competency  Management  &  Assessments 

4 

Workforce  Planning  -  Pilot  Program 

5 

Retention  and  Recognition  Incentives 

6 

Career  Broadening  and  Academic  Programs 

7 

Intern  Programs 

8 

Recruiting  Incentives 

9 

Outreach  Programs 

10 

Journeyman  Hiring 

1 1 

Highly  Qualified  Experts  (HQE's) 

21 


Tomorrow’s  T&E  Workforce 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  Size  -  About  the  same  total  number 

•  Composition  -  Divided  along  the  same 
Component/Agency  Lines 

•  Age  -  Younger 

•  Retention  -  Current  economy  will 
slowdown  retirements 

•  Competency  -  More  Focused 
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Education  &  Training 


DEVELOPMENTAL  TEST  &  EVALUATION 


Common  acquisition 
foundation 
knowledge  and  skills 


Career  Field 
foundation 
knowledge  and  skills 


“Plus”  or  job  competency 
point-of-need 
training  (frequently  CLMs) 
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Core  Plus  Guide 


DEVELOPMENTAL  TEST  &  EVALUATION 


Type  of 
Assignment 

Core  Certification 
Standards 


Unique  Position 
Trsiining 

Standarcm 


Core  Plus 
Development  Guide 

Footnotes 


Program  Management  (A 


Type  of  Assignment 

Representative  Activities 

_ - — — '  '  "  "  Weapon  Systems 

IPT  lead  &  oversight,  pre-award  contract  matters,  financial  management,  risk  management,  systems  engineering, 

total  ownership  cost,  contract  coordination,  &  communications.  As  appropriate,  coordination  on  net-centric  technology 
as  a  key  enabler  that  allows  a  C2  capability  to  operate  in  the  joint  operational  environment  and  force  protection  with  a 
minimally  sized  forward  footprint;  or  oversight  and/or  management  of  significant  segments  of  a  DoD  Space  MDAP. 

Services 

DoD  professional,  admin  &  mgmt  support  service  contracting  as  relates  to  developing  clearly  stated  &  actionable 
requirements  packages,  coordination  with  local  Procurement  Contracting  Officers  and  ensuring  opportunities  for 
socio-economic  business  concerns.  Performs  all  Acquisition  Strategy  Requirements  actions  noted  in  Attachment  1  to 
AT&L  Services  Memo  of  10/2/06. 

Business  Mgt  Systems/IT 

Transformation  integration,  planning  and  performance,  and  investment  management  as  applies  to  the  acquisition 
community,  program  office(s)  &  system(s)  under  development. 

International  Acquisition 

Groundwork  for  future  cooperation  during  pre-system  acquisition  or  participated  in  successful  cooperative 
development  or  production  partnerships  during  system  acquisition  with  allied  &  friendly  foreign  nation.  Participation  in 
successful  cooperative  development  or  production  partnership  during  pre-system  acquisition  or  system  acquisition 
with  allied  &  friendly  foreign  nations.  Other  type  of  assignment  also  applies. 

Core  Certification  Standards  1 


Acquisition 
Training  2 


None  Required 


PMT  352A:  Program  Management  Office  Course 
PMT  352B:  Program  Management  Office  Course  CR 

SYS  202:  Intermediate  System  Planning,  Research,  Development,  &  Engineering  (Required  for  certification  on  4/1/08) 
Formal  education  not  required  for  certification. _ 

4  years  acquisition  experience  with  at  least: 

•  2  years  in  a  program  office/similar  organization  (dedicated  matrix  support  to  a  PM,  PEO,  DCMA  Program  Integrator, 
or  Supervisors  of  Shipbuilding) 

•  1  year  in  a  program  management  position  with  cost,  schedule,  and  performance  responsibilities 


PEOs;  PM/DPM  of  MDAP/MAIS;  PM/DPM 
of  Significant  Non  Major  Programs 


Unique  Position  Training  Standards 


PMT  401:  Program  Mgr's  Course  CR  and  PMT  402:  Executive  Program  Mgr’s  Course  CR; 
OR  PMT  302:  Advanced  PM  Course  and  PMT  402:  Executive  Program  Mgr’s  Course  CR 


Core  Plus  Development  Guide 


-  Training2 

Weapon 

Systems 

Services 

Business 
Mgmt,  IT 

International 

ACQ  452:  Forging  Stakeholder  Relationships  CR 

X 

X 

X 

BCF206  \  Cost  Risk  Analysis  CR 

X 

X 

X 

BCF  207:  Economic  Analysis  CR 

X 

X 

X 

BCF  209:  Acquisition  Reporting  for  Major  Defense  Acquisition  Programs  CR 

X 

X 

IRM  201:  Intermediate  Information  Systems  Acquisition  CR 

X 

X 

X 

LOG  200:  Intermediate  Acquisition  Logistics,  Part  A 

X 

X 

LOG  204:  Configuration  Management 

X 

X 

LOG  235:  Performance  Based  Logistics,  Part  A 

X 

X 

LOG  236:  Performance  Based  Logistics,  Part  B  CR 

X 

X 

PMT  304:  Advanced  International  Management  Workshop  CR 

X 

PMT  403:  Program  Manager’s  Skills  (ACAT  III  only)  CR 

X 

X 

X 

PQM  201A:  Intermediate  Production,  Quality  &  Manufacturing,  Part  A 

X 

SAM  301:  Advanced  Software  Management  CR 

X 

X 

X 

SYS  203:  Intermediate  SPRDE  Part  B  CR 

X 

TST  202:  Intermediate  Test  &  Evaluation  CR 

X 

CLC  032:  Understanding  &  Utilizing  Performance  Based  Payments 

X 

X 

CLE  008:  Six  Sigma;  Concepts  and  Processes 

X 

X 

X 

CLL  201:  Diminishing  Mfg  Sources  &  Materials  Shortages  Fundamentals 

X 

X 

X 

Education 

Type  Assignment 


At  least  24  semester  hours  from  among  accounting,  business  finance,  law,  contracts,  purchasing,  economics,  industrial  management,  marketing, 
quantitative  methods,  and  organization  and  management  (DANTES  equivalency  may  be  substituted). 


Experience 


2  additional  years  of  acquisition  experience 


L  These  Standards  list  the  training,  education  and  experience  required  for  certification  at  this  level. 

2  A  “CR”  following  a  course  title  indicates  the  course  is  delivered  as  resident  based  instruction. 

3  Workforce  members  assigned  to  the  position(s)  identified  must  meet  the  training  standard(s)  identified  within  six  (6)  months  of  assignment. 

4  When  preparing  your  I  DP,  you  and  your  supervisor  should  consider  the  training,  education  and  experience  listed  in  this  and  the  l®\gt  level  Core 
Plus  Developmental  Guides  if  not  already  completed. 


DEVELOPMENTAL  TEST  &  EVALUATION 


Summary 


In  Summary 

The  T&E  Acquisition  Work  Force  has 
and  will  continue  to  adjust  and  learn 
new  skills,  as  necessary,  to  meet  the 
product  and  system  acquisition 
challenges,  and  provide  accurate  and 
complete  performance  and  risk 
assessments  decision-makers  require 
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DEVELOPMENTAL  TEST  &  EVALUATION 


Ms.  Darlene  Mosser-Kerner 

703-697-3406 
DSN:  227 

Darlene. mosser-kerner@osd.  mil 

Visit  Our  Website 
h  ttp://www.acq.  osd.mil/sse/dte/ 
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DEVELOPMENTAL  TEST  &  EVALUATION 


BACK-UP  CHARTS 
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SEC  852  Initiative  Focus  Areas 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  Line  Item  1:  Training  Enhancement  and  Capacity  Expansion. 

•  Initiatives  in  this  line  item  will  provide  additional  throughput, 
curriculum  development  and  other  learning  support  to  meet 
previously  unmet  demand.  There  is  a  gap  between  the 
Components’  demand  for  certification  and  assignment-specific 
training  and  the  capacity  of  the  Defense  Acquisition  University  to 
provide  training.  In  addition,  the  Components  have  identified  unique 
Component  requirements  that  will  be  funded. 

•  Line  Item  2:  Comprehensive  Acquisition  Workforce  and 
Student  Information  System. 

•  Initiatives  in  this  line  item  will  provide  a  single  clearinghouse  for 
workforce  data,  the  statutorily  mandated  workforce  management 
information  system,  and  a  commercial  best-in-class  student 
information  system.  The  objective  is  to  drive  standardization, 
integrate  systems,  build  transparency,  and  improve  data  quality,  and 
ensure  a  comprehensive  workforce  analysis  capability.  This  will 
enable  strategic  workforce  planning  and  decision  making  capability. 
This  line  item  will  ensure  current,  accurate  and  transparent 
information  is  available  on  the  Defense  acquisition  workforce  for 
acquisition  leaders. 
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SEC  852  Initiative  Focus  Areas 


•  Line  Item  3:  Competency  Management  and  Assessments. 

•  Initiatives  in  this  line  item  will  provide  a  standardized  competency 
management  program,  validated  competency  models  for  all  career 
fields,  tools  to  produce  individual  development  plans,  the  ability  to 
determine  training  needs,  and  support  for  human  capital  planning. 
The  Department  is  committed  to  an  enterprise  competency 
management  and  workforce  assessment  capability.  This  will 
improve  the  Department’s  ability  to  appropriately  identify  workforce 
skill  gaps,  requirements,  and  needed  learning  assets. 

•  Line  Item  4:  Workforce  Planning  Pilot  Program. 

•  This  Air  Force  pilot  initiative  is  being  worked  with  OPM  to  develop  a 
human  capital  architecture  that  includes  interview  tools, 
occupational  questionnaires,  and  job  previews.  If  successful,  it  has 
the  potential  to  enable  acquisition  centers  to  deploy  competency- 
based  tools  and  create  organization  specific  recruitment  and 
retention  strategies.  A  key  outcome  will  be  tailored  succession 
plans  that  will  help  acquisition  organizations  transition  from  their 
current  state  to  their  forecasted  “to  be”  mission. 
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SEC  852  Initiative  Focus  Areas 


•  Line  Item  5:  Retention  and  Recognition  Incentives. 

•  Initiatives  in  this  line  item  will  retain  high  performers  with  critical 
skills  and  in  key  leadership  positions  and  improve  retention  in 
positions  that  are  in  short  supply  through  incentives  and  programs 
designed  to  make  a  career  in  DoD  acquisition  more  attractive.  The 
forecasted  loss  of  corporate  knowledge  and  expertise  has  the 
potential  to  significantly  impact  the  ability  of  the  acquisition 
workforce  to  carry  out  its  mission  of  achieving  successful  acquisition 
outcomes. 

•  Line  Item  6:  Career  Broadening  and  Academic  Programs. 

•  Initiatives  in  this  line  item  will  provide  developmental  assignments, 
rotations,  programs  at  academic  institutions,  training  outside  one’s 
current  specialization,  and  opportunities  to  acquire  joint  and 
interagency  experience.  The  legacy  career  structure  for  civilian 
employees  has  emphasized  depth  over  breadth,  which  results  in  a 
“silo”  framework  that  inhibits  the  broader  perspective  needed  to 
manage  complex  acquisition  programs.  A  more  attractive  career 
structure,  which  will  improve  retention  in  the  acquisition  workforce,  is 
one  that  provides  breadth  of  experience,  training  and  education. 
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SEC  852  Initiative  Focus  Areas 


•  Line  Item  7:  Intern  Programs. 

•  Initiatives  in  this  line  item  will:  1)  recruit,  hire  and  develop  interns  to 
be  better  qualified  at  the  point  of  migration  into  the  acquisition 
workforce;  and  2)  augment  the  current  workforce  in  numbers  above 
the  currently  programmed  levels.  Seventy-three  percent  of  the 
current  Defense  acquisition  workforce  are  in  the  Baby  Boomer  and 
Traditional  generations  and  19  percent  (20,000  civilians)  are  eligible 
for  full  retirement.  At  the  same  time,  there  are  skill  and  competency 
areas  within  the  current  workforce  that  have  too  few  people  and 
need  to  be  built  up. 

•  Line  Item  8:  Recruiting  Incentives. 

•  Initiatives  in  this  line  item  seek  to  attract  qualified  applicants  in  such 
critically  needed  areas  as  Science,  Technology,  Engineering,  and 
Mathematics  (STEM),  minority  applicants,  and  recent  college 
graduates.  This  complements  other  hiring  initiatives  by  enabling 
DoD  to  more  effectively  hire  high  demand  talent  in  an  increasingly 
competitive  labor  market.  Hiring  incentives  include  first-duty-station 
PCS  moves,  hiring  bonuses,  student-loan  reimbursement,  tuition 
assistance,  scholarships,  and  SCEP  (formerly  Co-op)  programs. 
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SEC  852  Initiative  Focus  Areas 


DEVELOPMENTAL  TEST  &  EVALUATION 

•  Line  Item  9:  Outreach  Programs. 

•  Initiatives  in  this  line  item  will  deliberately  market  the  DoD  acquisition 
community  as  an  employer  of  choice.  It  is  a  DoD  goal  to  maintain  a 
diverse,  capable,  and  ready  civilian  and  military  workforce,  which  this  line 
item  will  support.  According  to  the  Gallup  poll  conducted  for  the  Council  for 
Excellence  in  Government,  DoD  is  one  of  the  most  attractive  brand  names 
in  the  federal  government.  However,  the  acquisition  community  needs  to 
be  more  active  in  reaching  out  to  prospective  employees.  A  preferred 
approach  is  to  leverage  federal,  DoD  P&R,  and  Component  programs  to 
improve  the  supply  of  quality  candidates. 

•  Line  Item  10:  Journeyman  Hiring  Programs. 

•  Initiatives  in  this  line  item  will  target  experienced  employees,  such  as 
retiring  military  and  seasoned  industry  candidates.  Workforce  analysis 
indicates  a  need  for  mid-career  hires  to  complement  our  intern  initiatives 
and  to  ensure  strong  bench  strength  to  fill  senior  and  executive  positions  as 
the  Baby  Boomer  and  Traditional  generations  depart  the  workplace. 

•  Line  Item  11:  Hiring  Expert  Knowledge  -  Highly  Qualified  Experts 
(HQE). 

•  Initiatives  in  this  line  item  are  to  hire  temporary  employees  for  up  to  five 
years.  In  many  cases,  there  is  a  need  for  people  with  special  expertise  who 
are  already  at  a  senior  level  and  are  recognized  experts  in  an  acquisition 
field  or  related  discipline.  HQEs  are  hired  under  a  special  hiring  authority 
granted  by  Congress.  This  line  item  complements  other  hiring  initiatives. 


Extending  Test  and  Evaluation  Modeling 
and  Simulation  Capabilities  with  Gaming 

Technology 

NDIA  National  Test  &  Evaluation 
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Operational  Test  Command 


Agenda 


•  Purpose 

•  Current  Efforts 

•  TTEC  Integration  Facility 

•  Gaming  Technology 

•  Simulation  and  Gaming 

•  Cost  Benefit 

•  Future  Work 

•  Summary 
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Operational  Test  Command 


Purpose 

To  provide  information  on  the  collaborative  efforts 
between  OTC  and  TRADOC  to  integrate  gaming 
technology  into  OTC’s  test  technology  enterprise  to 
provide  more  capable  simulation  and  visualization  tools 
for  use  by  the  Testing  and  Evaluation  community. 
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Operational  Test  Command 


Current  Efforts 


•  Technology  Transformation  Division  (TTD)  of  the  US 
Army  Operational  Test  Command  (USAOTC). 

-  Federates  models  and  simulations  to  form  a  tool  kit  of 
Enterprise  Members  (OASIS)  for  use  by  Test  Officers 
and  Operational  Research/System  Analysts 

-  Integrates  tools  from  various  battlefield  areas  to 
provided  an  adjustable  set  that  maximizes  the 
strengths  of  each  while  minimizing  any  shortcomings  of 
the  individual  simulations. 

•  TRADOC  Capabilities  Manager  (TCM)  Gaming. 

-  Manages  gaming  technology  fielded  to  units  for  use  in 
training. 

-  Provides  terrain  and  entity  data. 
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Operational  Test  Command 


History 


•  OTC  has  over  20  years  of  experience  collaborating  with 
the  training  domain  to  share  knowledge  and  costs  to 
economiccally  federate  simulations 
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Operational  Test  Command 


OASIS 

OY-1:  OTC's  Advanced  Simulation  and  Instrumentation 
Systems  (OASIS)  Technology  Enterprise  in  Support  of  the  Test 
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Operational  Test  Command 


Gaming  Technology 

•  Off  the  shelf  simulation  developed  by  industry  . 

•  Ability  for  individual  and  small  unit  leaders  to  act  in  a  first 
person  fashion  under  battlefield  conditions  and  refine  their 
Tactics,  Techniques  and  Procedures  (TTP). 

•  Training  tool  with  Test  and  Evaluation  applications. 

•  Visualization  tool  for  the  unit  under  test  and  for  the  test 
officer. 
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Operational  Test  Command 


Test  Technology 
Execution  Center 
Integration  Facility 


•  Works  to  keep  simulations  updated  and  exchanging  data. 

•  Brings  simulations  into  common  language;  i.e.  Distributed 
Interactive  Simulation  (DIS),  High  Level  Architecture 
(HLA),  MATREX. 

•  Links  technology  provided  by  TRADOC  TCM  Gaming  to 
OTC  tools. 
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Operational  Test  Command 


Constructive  View 
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Operational  Test  Command 


Updated  Constructive 

View 
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Operational  Test  Command 


Gamer’s  View 


Operational  Test  Command 


Cost  Benefit 

•  No  development  costs. 

•  Minimal  integration  cost. 

•  Test  units  already  know  the  gaming  simulation 

•  Much  more  credible  set  of  observations  than  previously 
available  to  those  watching  the  simulation. 

•  Augment  simulation  quantitative  data  with  more  valuable 
qualitative  data  for  eventual  use  by  the  evaluation  team. 


Operational  Test  Command 


Future  Work 

•  Utilize  latest  gaming  from  TRADOC  TCM  after  next 
selection  decision. 

•  Adapt  to  latest  US  Army  simulations  when  fielded. 

•  Continue  use  of  latest  integration  language. 
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Operational  Test  Command 


Summary 

Gaming  technology  can  provide  a  powerful  solution  to 
some  gaps  when  federated  with  existing  simulations  and 
data  collection  systems  used  within  the  Test  and 
Evaluation  Community. 
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Operational  Test  Command 


I  DA 


An  Enhanced  Analysis  of 
Alternatives  (AoA) 

A  Mission-Oriented,  Evaluation-Based 
Framework  for  Defense  Test  &  Evaluation 

NDIA 

March  2-5.  2009 


Vince  Roske 

Institute  for  Defense  Analyses 

vroske@ida.org 

703  575  6632 
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IIX4  Role  of  the  Analysis  of  Alternatives 

(AoA) 

Analysis  of  Alternatives  (AoA) 

Translates  tasks  be  performed  in  an  operational  context 
into  a  system  “solution”  for  acquisition 


MSA 


MS  B 


MS  C 


“The  evaluation  of  the  performance,  operational  effectiveness,  operational  suitability, 
and  estimated  costs  of  alternative  systems  to  meet  a  mission  capability.  ....The  AoA  is 
one  of  the  key  inputs  to  defining  the  system  capabilities  in  the  Capability  Development 
Document  (CDD)” 


Insights  Needed  to  Build  a  TEMP 
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I  DA 


AoA  Methodology 


MSA 


MS  B 


MS  C 


Full  Rate 
Prod  DR 


Strategic 

Joint 

Guidance 

Concepts 

OSD/JCS  COCOM 


Engineering  & 
Manufacturing 
Development  & 
Demonstration 


CPD 


Production  & 
Deployment 

o 


Incremental  Development? 


CBA 


EVALUATION  FRAMEWORK 


TASKS 

& 

CONDITIONS 


AoA  Performed  by  a 
JROC  Designated 
(Service)  Sponsor 


ICD  IcapabilitTX 

NEED  1 


Solution 

Alternatives 


Solution  Descriptions 

‘Performance  Characteristics 
•Development  &  Testing 
•Acquisition  Alternatives 


MS  B 


►Procurement  Specifications 


o&s 


> 


I  DA 


A  Review  of  AoAs 


•  Reviewed  AoAs 

-  AoAs  Done  Between  2003  and  2008  AND  Done  for  DOT&E  Oversight  List 
Programs 

•  Implications  for  T&E: 

-  Each  Contains  a  Description  of  an  Operational  Context  Potentially  Useful  to  T&E 
Planning 

•  Scenario,  Forces,  Objectives  CONOPS,  Climate,  OPTEMPO,  etc 

-  Each  includes  a  “Sensitivity  Analysis”  (SA)  Relating  System  Performance  to 
Mission  Accomplishment 

HOWEVER:  SAs  do  NOT  Relate  Variations  in  System  Characteristics 
Performance  to  Mission  Effectiveness 

•  SAs  Treat  Performance  Characteristics  as  “Fixed”; 

•  Used  to  Derive  System  Fleet  Size  or  System  Configurations  of  Fixed 
Components  for  “Mission  Accomplishment”  Across  Varied  Scenarios 

-  AoA  Informs  a  System  Acquisition  Strategy 

»  A  “Packing  Algorithm”  Paradigm:  “How  Many  “Systems  as  Defined” 
are  Needed,  When,  for  Mission  Success  and  at  What  Cost 

-  AoA  Does  NOT  Recognize  T&E  as  a  Customer 

»  T&E  Examines  System  and  Component  Characteristics 
Performance  relative  to  Mission  Effectiveness  measures 
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Analysis  of  Alternatives  (AoA) 
Requirements-Based 


EVALUATION  FRAMEWORK 
•Fixed  Performance  Characteristics 
•  VARY  Fleet  Size,  Mix,  or  System  Composition 
•To  Operate  in  the  Force  Structure 
-DOTLPF,  CONOP,  OPTEMPO 
•To  Accomplish  the  Specified  Missions. 
•Across  the  Scenarios 


CAPABILITY 
NEED 


Solution 

Vk  ^  >  Solution  Descriptions 

Alternatives 

SENSITIVITY  ^  a  • 

ama.vcc  ‘Acquisition 

£> 


-Physical  Description 
-Suitability  Characteristics 
-System  Performance 


-Fleet  Size  &  Cost 
-Procurement  Schedule 


CDD 


MS  B 


AoA  Performed  by 
JROC  Designated 
(Service)  Sponsor 
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An  Enhanced  Analysis  of  Alternatives  (AoA) 
Capability-Based  Sensitivity  Analysis 


EVALUATION  FRAMEWORK 
•VARY  the  System  Characteristics 
In  the  Sensitivity  Analysis 
-Develops  the  “Performance, 
Effectiveness,  Cost”  Trade  Space 
•To  Operate  in  the  Force  Structure 
-DOTLPF,  CONOP,  OPTEMPO 
•To  Accomplish  the  Specified  Tasks 
•Across  the  Scenarios  &  Conditions 


AoA 


JCIDS 

CAPABILITY 

NEED 


SCENARIO  (s) 


ICD 

Tasks 

& 

Conditions 

•Force  Structure 
• CONOP 
•OPTEMPO 


OPERATIONAL  CONTEXT 


Solution 

V  \/L^  J 

_ xX-' _ ! > 

Solution  Descriptions 

Alternatives 

SENSITIVITY 

AKIAI  VCIC 

•System  Description 

CDD 


o 


MS  B 


•VARY  Characteristics 
-Physical 
-Performance 

AoA  Performed  by 
JROC  Designated 
(Service)  Sponsor 


-Mission  Effective  System  Characteristics 
Performance  Requirements 
-Context  &  Conditions  for  Testing 
-Development  RFP  Requirements  and 
Grading  Criteria  (Sections  L  &  M) 
Acquisition 
-Fleet  Size  &  Cost 
-Procurement  Schedule 


I  DA 


Enhanced  AoA 

Treats  System  Performance  Characteristics 
as  Variables  in  the  AoA  /  SA 


Enhanced  AoA 
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I  DA 


An  Enhanced  AoA  an  Integrating 

Process 


SUITABILITY  (Rqmts  &  Evaluation  Framework) 


CAPABILITY 


Early  Testing  of 
Assumptions 
& 

Representations  (M&S) 
Pre  Milestone  A 


Procurements 

Alternatives 

Acquisition  Strategy 

RFP  Content 
Sections  L&M 
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DA 


T&E  Benefits  from  an  Enhanced  AoA 

•  Enhanced  Description  of  the  Operational  Context 

-  For  use  in  the  AoA 

-  For  deriving  operational  environments  for  testing 

-  For  integrating  development  and  operational  test  planning 

•  Consistent  Validated  Evaluation  Framework 

-  For  systems  design,  testing,  and  assessment  of  system  and  mission  cost  /  effectiveness 

-  MDA  better  informed  for  Confidence  in  the  System  and  for  mitigating  any  performance 
deficiencies 

•  More  Comprehensive  and  Effective  Test  Planning 

-  Enhancing  Confidence  in  the  AoA 

•  Early  testing  of  key  AoA  assumptions  and  representations 

-  Responsive  to  pre-Milestone  A  program  reviews 

-  Tasking  and  Resource  Planning  in  the  TEMP 

»  For  M&S  in  the  AoA 
»  For  M&S  supporting  DT  and  OT  events 

-  Mission  Oriented,  Evaluation  Based  Test  Planning 

•  Testing  integrated  with  the  AoA 

•  Testing  focused  on  most  important  performance  characteristics  and  conditions 

•  Test  results  evaluated  for  mission  effectiveness  in  the  AoA  analysis  framework 


Harmonized  Acquisition  Milestone  Decision  Support 

System  Design,  Cost  /  Effectiveness  Evaluation,  Acquisition  Strategy,  and  Test  Program 
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I D A  Benefits  to  T&E:  Capability  Test  Method  (CTM) 


....System.^  ACq 
Design 


AoAP 


Approved  AoA 
Plan 


Analytical  Baseline 
(DPS,  MSFD,  FYAB) 


Exploratory  Analysis 
Results 


/  ClJTL 
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UJTL 

J 

JOpsC 


J  OpsC  Family 
(J  OC,  JFC,  JIC) 


OTF 


Other  Test  Plans 
(DT,  OT,  LF) 


J  oint  Capabilities 
Document  (J  CD) 


I  nitial  Capabilities 
Document  (I CD) 


DOTMLPF  Change 
Request  (DCR) 


CDD 


Capability  Developi 
.  Document  (CDD) 
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CPD 


/JCA 
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J  oint  Capability 
Areas 

Develop  T&E  Strategy 

▼  51 

-c 

^  w 

Capability  Production 
Document  (CPD) 


•Who  and  how?  Fair  to  T&E  or  others? 
•A  “parallel”  to  acquisition  process? 
•Performance  requirements? 
•Evaluation  framework? 

•How  to  synchronize? 


TESTEMP.J  OCT  h 

J  oint  Operational 
Context  for  Test 
(JOC-T) 


A 

Test  Plan 

J 
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I  DA 


Benefits  to  T&E: 

An  Emerging  TEMP  Format 


Part  I  Part  II  Part  III 

Introduction  Test  Program  Test  &  Evaluation 

Management  &  Schedule  Strategy 


Part  IV 
Resource 
Summary 


Brief  Purpose  of  TEMP 

Brief  Mission  Description 

Brief  System  Description 
-System  Threat  Assessment 
-Program  Background 
-Key  capabilities 
-Key  Interfaces 
-Cert  Rqmts 
-Sys  Engr  Rqmts 


T&E  Management 

Comment  T&E  Data 
Base  Rqmts 

Deficiency  Reporting 

TEMP  Updates 

Integrated  Test  Proc 
Schedule 


•AoA  Overview  (T&E  Perspective) 
-Operational  Context  of  Evaluations 
-AoA  System/Mission  Evaluation  Approach 
’Acquisition  Strategy  Overview 
-Milestone  Information  Needs 
-System  Maturity  Schedule 


Describing  a 
Mission-Oriented, 
Evaluation-Based 
Test  Program 


•System  Evaluation  Strategy  (T&E 
Perspective,  Relates  to  AoA,  Explains 
Testing’s  Role  in  System/Mission 
Effectiveness  Evaluation) 

-  Joint  Operational  Context  for  Evaluations 

-  Evaluation  Approach 

-Mission  Effectiveness  Evaluations 
-Comparison  to  Current  Capability 
-Test  Support  to  Evaluations 
•Test  Strategy  (Testing’s  Approach  to 
Supporting  the  Evaluation  Strategy) 

-Joint  Operational  Context  for  Test  (JOC-T) 
-DT  &  OT  Objectives  from  Evaluation 
Strategy 

-Assumptions  and  M&S  Validation  Tests 
-Integrated  DT  &  OT  Planning  Objectives 

•Developmental  Evaluation  Approach 
-Developmental  Test  Approach 
-M&S,  Test  Limitations 
•Live  Fire  Evaluation  Approach 
-Test  Objectives,  M&S,  Test  Limitations 
•Certification  for  IOT&E 
•Operational  Evaluation  Approach 
-Operational  Test  Approach 
-M&S,  Test  Limitations 
•Other  Certifications 
•Reliability  Growth 
•Future  T&E 


Introduction 
-Test  Articles 

-Test  Sites  &  Instrumentation 
-Test  Support  Equipment 
-Threat  Representations 
-Test  Targets  &  Expendables 
-OPFOR  Test  Support 
-M&S  and  Test-beds 
-Joint  Operational  Test 
Environment 
-Special  Requirements 

Federal,  state,  Local  Rqmts 

Manpower/  Personnel  Training 

Test  Funding  Summary 
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Realities  Facing  the  World  of 
Defense  and  Homeland  Security 

Test  &  Evaluation 

Maj  Gen  Steve  Sargeant 
Commander,  AFOTEC 

3  March  2009 

NDIA  Test  &  Evaluation  National  Conference 
Atlantic  City,  NJ 


"This  information  is  approved  for  public  release;  distribution  is  unlimited." 


Purpose 


*  Describe  AFOTEC’s  efforts  to  enhance  acquisition 
excellence 


2 


W  Overview 

*  Education  and  Training 

*  Common  Lexicon 

*  Institutionalized  Early  Influence  and  Integrated  DT/OT 

*  Way  Ahead 
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Education  and  Training 

*  Partnerships  with: 

-  Defense  Acquisition  University 

■  AFOTEC  guest  lecturers  integrated  into  select  courses 

■  Provided  case  studies  for  T&E  courseware  based  on  Section 
231  initiatives 

-Air  Force  Institute  of  Technology 

■  Instruction  at  AFOTEC  T&E  University  Reliability  Courses 

■  AFIT  T&E  Certificate  in  work 

*  AFMC  Wing  and  Group  Commander’s  Courses 

*  Identifying  and  tracking  T&E  experience 
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Common  Lexicon 


w 

«> 

*  Commonly  understood 

-  Early  Influence 

-  Integrated  DT/OT 

*  Definitions  still  in  refinement 
-Test  plans 

-  Operationally  relevant  data 

-  Production  representative 

-  Operationally  realistic  scenarios 
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W  Institutionalizing  El  and  IDT/OT 

*  Early  Influence 

-  Refining  test  requirements  (M&S)  early,  enabling  correction 
of  shortfalls  when  costs  are  lower 

*  Integrated  DT/OT 

-  Contracts  with  access  to  DT  data 

-  Mold  DT  scenarios  to  gather  operationally  relevant  data 

-  Communication  and  coordination  required  to  manage  risk 

*  Reducing  the  cost  &  time  of  the  T&E  enterprise 

-  Maintain  the  level  of  confidence 

-  Get  capabilities  to  the  warfighter  on  time  and  at  cost 
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LJDAM  OUE  IDT/OT 


Initial  test  design 

Mar-May  07  July  07 


19  Feb-21  Mar 


1  May 


30  May  08 


- >  ~ 

Requirements 

&  Direction  AFOTEC 
Involved 


SDD 


OUE  Final 
Report 


II  I  V  vy  I  V 

-  Initial  OT  requirement  was  10  weapon  drops 

-  Initial  OUE  cost  and  schedule  estimate  was  S2.58M  /  31  Days 


Fielding 

Decision 


*  Actual  test  execution  with  IDT/OT  philosophy 

28  Mar  18  Apr 

Mar-May  07  July  07 


19  Feb-7  Mar 


6  May  08 


Requirements 

&  Direction  AFOTEC 
Involved 


SDD 


OUE 

Status  gelding 

Rep0rt  OUE  Final 

Report 


-  Leveraged  12  DT  weapon  drops,  reduced  OT  reqt  to  5 

-  Final  OUE  saved  $1.43M  and  accelerated  fielding  6  weeks 
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W  Way  Ahead 

*  Refine  and  codify  IDT/OT 

*  Continue  to  develop  a  common  understanding  of  IDT/OT 
process 

*  Focus  on  Cyberspace  Testing 
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Summary 


*  We  must  manage  of  T&E  experience 

*  Integrated  DT/OT  is  the  way  ahead  and  will  require 
breaking  paradigms  and  engender  greater 
communication  and  coordination  between  DT  and  OT 


AFOTEC’s  objective:  deliver  warfighting  capabilities  faster  and  with 
more  confidence ,  enabling  Airmen,  and  often  our  Joint  and  Coalition 
partners,  to  accomplish  their  mission  more  effectively  and  with  less  risk 
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W  Panel  Questions 

*  Army  Test  &  Evaluation  Command 

-  As  the  largest  test  agency  in  the  DoD,  how  will  you  recruit 
and  develop  your  workforce? 

*  Commander,  Operational  Test  &  Evaluation  Force 

-  What  is  the  effect  of  DoDI  5000.02  on  test  &  evaluation  of 
ship  building? 

*  Marine  Corps  Operational  Test  &  Evaluation  Agency 

-  Describe  your  ability  to  execute  OT&E  in  a  reduced 
resource  environment. 

*  Department  of  Homeland  Security 

-With  the  22  agencies  consolidated  under  the  DHS  umbrella 
and  reduced  budgets,  do  you  see  an  opportunity  for  an 
operational  test  agency  across  all  DHS  agencies  and  an 
integrated  approach  for  OT&E? 
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GAO 

Accountability  *  Integrity  *  Reliability 


What  Will  It  Take  to  Get 
Better  Program  Outcomes? 


25th  Annual  Test  &  Evaluation  National  Conference: 
“New  Administration,  New  Opportunities” 

March  2-5,  2009 


Paul  Francis,  Director 
Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 

francisp@gao.gov 
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Weapon  System  Investment  Levels  M.  if.  A,Q 


FY  2008  Dollars  in  Billions 


□  Expended  ^Outstanding 


Source:  GAO  analysis  of  DOD  data. 
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a  n  A  O 

Decline  in  Cost  and  Schedule  Outcomes 


50% 


40% 


30% 


20% 


10% 


0% 


Total  Acquisition  Costs  RDT&E  Costs  25%  Cost  Growth 


□  FY  2000  □  FY  2005  EFY2007 


Months 

25  1 
20- 

15- 

10- 

5- 


16 


21 


17 


Average  schedule  delay  in  delivering  initial  capabilities 


Source:  GAO  analysis  of  DOD  data. 


■  FY  2000  □  FY  2005  OFY  2007 
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A  GAO 

Accountability  *  Integrity  ♦  Reliability 

Cost  and  Schedule  Overruns  in  Five  Programs 


Total  Cost 
(billions  of  $) 

Planned  Latest 

Total  Quantities 

Planned  Latest 

Increase  in 
Unit  Cost 

Initial  Delivery  of 
Capability 

Planned  Latest 

JSF 

203.0 

240.0 

2,866 

2,458 

38% 

2010 

2012 

FCS 

88.3 

128.5 

15 

15 

46% 

2010 

2015 

SBIRS  High 

4.4 

10.5 

5 

3 

300% 

2003 

2009 

EFV 

8.7 

13.5 

1,025 

593 

1 68% 

2006 

2015 

H-1 

Upgrades 

3.4 

8.3 

284 

284 

140% 

2005 

2008 

Source:  GAO  analysis  of  DOD  data 
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Consequences  of  Poor  Outcomes 


k  G  A  O 

Accountability  *  Integrity  *  Reliability 


Cost  Growth 


I  I 


Reduces  DOD’s  I  Means  less  funding 
buying  power  I  for  other  priorities 
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DOD  must  request  more  funding  to 
cover  cost  overruns,  make  trade¬ 
offs  with  existing  programs,  delay 
the  start  of  new  programs,  or  take 
funds  from  other  accounts 


Schedule  Delays 


i 


Critical  capabilities  not  provided 
to  warfighter  when  needed 


DOD  must  operate  costly  legacy 
systems  longer  than  expected,  find 
alternatives  to  fill  capability  gaps,  or 
go  without  a  capability 
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A  Knowledge-Based  Approach 
Is  Key  to  Good  Outcomes 


p-  GAO 

Accountability  *  Integrity  *  Reliability 


Knowledge 
point  1 - 1 

T 


Knowledge 
point  2 


Knowledge 
point  3 


Technology 

Development 

A 

Product 

Development 


Production 


Knowledge  Point  1:  At  milestone  B,  a  match  is  achieved  between  the  user’s  needs  and 
the  developer’s  resources.  Technology  maturity  is  demonstrated  and  preliminary  design 
is  achieved. 

Knowledge  Point  2:  At  critical  design  review,  the  product  design  demonstrates  its  ability 
to  meet  user  needs  and  is  stable.  Prototype  demonstration  that  design  will  meet 
requirements. 


Knowledge  Point  3:  At  milestone  C,  it  is  demonstrated  that  the  product  can  be  produced 
within  cost,  schedule,  and  quality  targets.  Full-up,  integrated  product  tested  in  relevant 
environment. 
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Immature  Technologies  Ripple 
Through  the  Development  Cycle 


p-  GAO 

Accountability  *  Integrity  *  Reliability 


Percent  of  Programs 


Development 

start 

2005  assessment 

2006  assessment 


DOD  design 


review 


Auction 
decision 


2007  assessment 

2008  assessment 


Less  than  20%  of  programs 
have  mature  technologies  at 
start 


Most  programs  do  not  have 
mature  technologies  at  CDR 


Many  programs  still  maturing 
technologies  into  production 

Cost  growth  for  programs  with 
immature  technologies  was 
44%  higher 


Only  10  percent  of  programs 
had  completed  PDR  at  start 


Source:  GAO  analysis  of  DOD  data. 
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Programs  Proceed  Through 
CDR  without  Design  Stability 


Drawing  Releases 

Percent 

100  i - 


80 


60 


40 


20 


0 


DOD  design  Production 

review  decision 


2005  assessment 

2006  assessment 
Source:  GAO  analysis  of  DOD  data. 


2007  assessment 

2008  assessment 


GAO 

Accountability  *  Integrity  *  Reliability 


Goal  is  90%  drawings 
releasable  at  CDR 


3/4  of  programs  do  not  meet 
this  standard  at  CDR 


At  milestone  C,  over  1/3  of 
programs  still  did  not  meet 
this  standard 
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i 


R&D  Cost  Growth  Experience 


GAO 

Accountability  *  Integrity  *  Reliability 


Percentage  of  RDT&E  cost  increase  over  development  estimate 

30 


Total  increase  28.3% 
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GAO 

Accountability  *  Integrity  *  Reliability 


Other  Observations  on  Current  Practices 


•  Fewer  than  half  of  programs  plan  to  test  fully-integrated,  production- 
representative  prototypes  before  Milestone  C  (including  JSF  and 
FCS) 

•  During  FY  2007,  DOT&E  reports  that  50%  of  programs  failed 
operational  suitability;  reliability  is  on  a  downward  trend 

•  GAO  has  recently  reported  that  during  FY  2008,  missile  defense 
assets  were  produced  and  fielded  before  being  flight  tested 

•  Programs,  like  JSF,  are  using  cost-reimbursable  contracts  in 
production. 
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New  Reforms 
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p-  GAO 

Accountability  *  Integrity  *  Reliability 


2008  DOD  5000  Policy 

•  Reinvigorated  Milestone  A  and  technology  development  phase 

•  Configuration  Steering  Boards  established  to  control  requirements  creep 

•  Stronger  emphasis  on  systems  engineering 

•  Preliminary  Design  Review  before  Milestone  B 

•  Formal  post-CDR  assessment 

•  Stronger  pre-milestone  C  requirements  (DT&E,  M&S,  production- 
representative  prototypes,  pilot  line  production) 


2009  Weapon  System  Acquisition  Reform  Act  (proposed) 

•  Identify  and  fill  gaps  in  systems  engineering  capabilities 

•  Create  Director  of  Developmental  Test  and  Evaluation 

•  DDR&E  review  and  assessment  of  critical  technology  maturities 

•  Create  Director  of  Independent  Cost  Assessment 

•  Cut  across  requirements,  budgeting,  and  acquisition  stovepipes  to  make 
needed  tradeoffs. 

•  PDR  before  Milestone  B 


Cause:  Process  Pressures 


GAO 

Accountability  *  Integrity  *  Reliability 


Promise  High 
Performance 


Promise  Low 
Resource  Demands 


>  Move  Forward, 
Get  Knowledge  Later 


The  process  is  not  broken:  it's  in  equiiibrium 
Good  people  are  not  put  in  a  position  to  succeed 
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Prognosis  for  Change 


GAO 

Accountability  *  Integrity  *  Reliability 


•  Weapon  system  issues  have  been  consistent  for  30  years 

•  They  are  primarily  not  due  to  mistakes,  lack  of  expertise,  or 
unforeseeable  events 

•  Consider  the  process  as  being  in  equilibrium  versus  broken: 

o  The  acquisition  process  may  be  producing  what  the 
participants  collectively  want  or  are  willing  to  settle  for. 

o  It  is  a  rational  process  that  involves  good  people.  It  works — 
this  is  how  programs  get  money  ancTsurvive. 

•  Our  principles  are  revealed  by  what  we  do  and  what  we  do  with 
money;  if  unexecutable  programs  continue  to  win  funds,  then  our 
principles  remain  something  other  than  what  is  stated  in  policy. 

•  Process  reforms,  funding  cuts,  and  cancellations  aren’t  enough  to 
change  the  culture  or  equilibrium:  programs  with  executable 
strategies  (technology,  design,  test,  &  cost)  must  win  the  budget 
battles. 

•  For  this  to  happen,  we  need  a  significant  emotional  event;  I  hope 
that  a  new  administration  and  new  opportunities  may  constitute 
that  event. 
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S©  RDECOM 


Integrating  the  Systems 
Engineering  "V"  in  a 
Systems  of  Systems 

3  March  2009 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

Mr.  Jack  Sheehan  (jhs@orsacorp.com) 

Dr.  Paul  H.  Deitz  (Paul.H.Deitz@us.army.mil) 

Dr.  David  M.  Bassan 


Bottom  Line  Up  Front 


•  Employ  a  Mission-based  SE  in  an  SoS  environment  approach  to  complete  the  (traditional) 
Materiel-based  SE  “V”  (SE-V)  in  the  development  of  DOTMLPF  Capabilities  for  the 
warfighter. 

•  Retaining  Mission,  Task,  and  Human  Dimension  context  throughout  the  Capability  lifecycle 
provides  assessment  results 

-  in  traditional  Materiel-base  terms  and 

-  their  relationships  and  contributions  both  direct  and  indirect 

to  warfighting  operational  performance  and  mission  effectiveness. 


•  Mission-based  SE  in  SoS  is  an  extension  of  MBT&E  compatible  with  the 

-  OSD/P&R  directives  for  reporting  METL-based  Readiness 

-  Joint  GEL  directive  for  certifying  deploying  units, 

-  JCIDS  Capability-based  Acquisition 

-  DoD  SE  Guide  for  SoS,  and 

-  DOT&E  JTEM  framework  and  procedures. 


METL:  Mission-Essential-Task-List  with  measures,  conditions  and  standards  to  accomplished  a  desired  end  result 

SoS:  Systems  of  Systems 

SE  “V”:  Top-Down  definition  and  design  then  Build  followed  by  Bottom-Up  integration,  and  verfication 

DOTMLPF:  Doctrine,  Organization,  Materiel,  Leadership,  Personnel,  and  Facilities 

FID:  Army  Fluman  Dimension  initiative  in  three  behavior  domains  --  Social,  Cognitive,  and  Physical 

MBT&E:  ATEC  Mission-based  Test  &  Evaluation  framework,  procedure,  and  complexity  constraint  strategies 

GEL:  Joint  Guidance  for  Employment  of  Forces  for  unit  certification  prior  to  deployment 

P&R:  OSD  Personnel  &  Readiness 

JTEM:  Joint  Test  &  Evaluation  Methodology 


HmcoM*  Definitions  ,i«f2) 


•  System  (S)*  -  A  functionally,  physically,  and/or  behaviorally  related  group  of 
regularly  interacting  or  interdependent  elements;  that  group  of  elements 
forming  a  unified  whole  [JP  1-02  &  JP  3-0]. 

•  Capability*  -  is  the  ability  to  achieve  a  desired  effect  under  specified 
standards  and  conditions  through  combinations  of  ways  and  means  to 
perform  a  set  of  tasks  [CJCS,  2007(2)]. 

•  Family  of  Systems  (FoS)*  -  a  set  of  systems  that  provide  similar 
capabilities  through  different  approaches  to  achieve  similar  or 
complementary  effects  [CJCS,  2007(1)]. 

•  System  of  Systems  (SoS)*  -  is  defined  as  a  set  or  arrangement  of  systems 
that  results  when  independent  and  useful  systems  are  integrated  into  a 
larger  system  that  delivers  unique  capabilities  [DoD,  2004(1)]. 

*  Taken  from  the  DoD  SE  Guide  for  SoS 


RDECOM 


Definitions  <2  of  2) 


•  Virtual  SoS*  -  Group  lacks  a  central  management  authority  and  a  centrally 
agreed  upon  purpose  for  the  system-of-systems. 

•  Collaborative  SoS*  -  Group  component  systems  interact  more  or  less 
voluntarily  to  fulfill  agreed  upon  central  purposes. 

•  Acknowledged  SoS*  -  Group  has  recognized  objectives,  a  designated 
manager,  and  resources  for  the  SoS;  however,  the  constituent  systems 
retain  their  independent  ownership,  objectives,  funding,  and  development 
and  sustainment  approaches.  Changes  in  the  systems  are  based  on 
collaboration  between  the  SoS  and  the  systems. 

•  Directed  SoS*  -  The  group,  an  integrated  system-of-systems,  is  built  and 
managed  to  fulfill  specific  purposes.  It  is  centrally  managed  during  long-term 
operation  to  continue  to  fulfill  those  purposes  as  well  as  any  new  ones  the 
system  owners  might  wish  to  address.  The  component  systems  maintain  an 
ability  to  operate  independently,  but  their  normal  operational  mode  is 
subordinated  to  the  central  managed  purpose. 


Taken  from  the  DoD  SE  Guide  for  SoS 


rdecom ^  Systems  Engineering  “V”  (iof3)  ■  I 


Concept  of  Operation: 


i 


Assessment 


k 


k 


k 


Opfralion^  ^  Maintenance 


Hi(|jb'Lev(l  Requirements 


Detailed  Requirements 


l 


i 


Sv  i  tern  Veri  lit  ,1  turn 


Definition 
and 

Decomposition 


A 


Subsystem  Wrifitatim 


Del  .tiled  Ocsi 


i  ii  Ee^r.ilitm  A,  Testing 


J  rii  p  ? !  fniv  ttUW  h  in 


/  Integration, 
Verification, 
^  /  and 


Validation 


Time 


Systems  of  System  Engineering  (2  of  3) 


INPUTS 


Taken  from  21  Jun2  20005  FCS  Review  to  DAB 


SE  in  SoS  Environment  Issues  (3  of  3) 


Taken  from  21  Jun2  20005  FCS  Review  to  DAB 


MBT&E  Building  Block 

Capability1  -  The  ability  to  achieve  a  desired  effect 
[or  result,  outcome,  or  consequence  of  a  task2]  ... 


-  under  specified  standards  and  conditions 

-  through  a  combination  of  means  and  ways 

-  to  perform  a  set  of  tasks. 


1. CJCSI  31 70.01  F,  May  2007 

2.  Taken  from  JP  1-02,  Mar  2007,  definition  of  effect. 


MBT&E  -  Task  Hierarchy 


MBT&E  Framework  —  v2 


Process/Products 


Commander’s  Task 
to  Subordinates 


- 1 - 

Transition  to  Allocating  Mission  Means 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


Commander’s  Task 
to  Subordinates 


- 1 - 

Transition  to  Allocating  SoS  Means 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


System  Attributes 


Transition  to  Allocating  Materiel  Means 

± 


Systems  Engineering 

•  Functional  Baseline 

•  Allocated  Baseline 

•  Product  Baseline 


f  AMSAA 

Excellence  in  Analysis 


Capability  =  Set  of  Tasks  +  Desired  Result 


Desired 
End  State 


Enables 


Operations  (Mission  Tasks) 

^ 

•  UJTLs 

Desired  Mission 

•  Service  TLs 

Task  Results 

•  Implied  Tasks 

Mission  Task  Capability 


System-of-Systems  Tasks 
Service  TLs 
Implied  Tasks 
Collective/Individual  Tasks 


T 


Desired  SoS 
Task  Results 


SoS  Task  Capability 


System  Performance 
•Functions  (shall  do) 
•  “shall  be’s” 

System  Performance 


Desired  System 
Performance  Results 


>- 


High  Level 
Tasks/Results 


Tasks/Results 
Specific  to 
System 


System 

Functions 


MBT&E  -  Task  Hierarchy 


MBT&E  Framework  —  v2 


Process/Products 


Commander’s  Task 
to  Subordinates 


- 1 - 

Transition  to  Allocating  Mission  Means 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


Commander’s  Task 
to  Subordinates 


Transition  to  Allocating  SoS  Means 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


System  Attributes 


Transition  to  Allocating  Materiel  Means 

± 


Systems  Engineering 

•  Functional  Baseline 

•  Allocated  Baseline 

•  Product  Baseline 


fAMSAA  | 

Excellence  in  Analysis 


Capability  =  Set  of  Tasks  +  Desired  Result 


Desired 
End  State 


Enables 


Operations  (Mission  Tasks) 

•  UJTLs 

•  Service  TLs 

•  Implied  Tasks 


Desired  Mission 
Task  Results 


Mission  Task  Capability 


System-of-Systems  Tasks 
Service  TLs 
Implied  Tasks 
Collective/Individual  Tasks 


Desired  SoS 
Task  Results 


T 


SoS  Task  Capability 


System  Performance 
•Functions  (shall  do) 

•  “shall  be’s” 

System  Performance 


Desired  System 
Performance  Results 


Essence: 
What  must  be 
accomplished 
and  Why 


Implementation 
DOTMLFP 
How  and  Who 


DOTMPF 

Component 


RDECOM 


Compatibility 


•  ATEC  Mission  Based  Test  &  Evaluation  is  consistent  with 

-  OSD  P&R  directives  for  reporting  METL-based  Readiness 

-  Joint  GEF  certification  for  Deployment  of  Operational  Forces 

-  JCIDS  Capability  based  Acquisition 

-  DoD  Systems  Engineering  guide  for  Systems  of  Systems  (SE  for  SoS) 

-  DOT&E  Joint  Test  &  Evaluation  Methodology  (JTEM) 


Based  on  Generic,  General  Purpose  Measures/Conditions/Standards 

-  TPFData  List  TO&E  (standing)  C-METL  OSD  P&R  Readiness 

-  TPFDeploment  Data  MTOF  (transient)  D-METL  Joint  GEF  Certification 

Based  on  Mission,  Situation  Specific  Measures/Conditions/Standards 


TPFDL:  Time  Phased  Force  Data  List  for  deployment  planning 

TPFDD:  Time  Phased  Force  Deployment  Data  for  deployment  execution 
TO&E:  Table  of  Organization  and  Equipment  for  a  standing  unit 

MTOF:  Mission  Task  Organized  Force  (modeling  &  simulation  term) 

C-METL:  Core  METL  defined  by  Army  Force  Generation  (ARFORGEN)  for  commonality  between  units  of  the  same  type 
D-METL:  Directed  METL  defined  by  Army  Force  Generation  (ARFORGEN)  for  a  unit  with  specific  deployment  orders 
GEL:  Joint  Guidance  for  Employment  of  Forces  for  unit  certification  prior  to  deployment 

P&R:  OSD  Personnel  &  Readiness  directives  for  METL-based  readiness  reporting 

JCIDS:  Joint  Capability  Integrated  Development  System 


MBT&E  Procedure 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  collect  information. 


understand  •  4  steps  to  understand  the  military  operations,  tasks,  task 
the  mission  capabilities  and  mission  context. _ 


0 


understand  •  2  steps  to  understand  the  components  and  attributes  of 
the  system  the  materiel  system-of-systems. _ 


•  1  additional  step  to  understand  the  mission  and  system 
linkages. 


o 


o 


design  the  •  7  steps  to  design  the  T&E  given  the  mission  and  system 
t&e _ understanding. _ 


determine  •  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
THE  RESULTS  data. 


REPORT 
THE  RESULTS 


1  step  to  format  and  report  the  results. 


Adaptation  to  Complete  SE-V 


•  N  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  collect  information. 


CD 


understand  •  4  steps  to  understand  the  military  operations,  tasks,  task 
the  mission  capabilities  and  mission  context. _ 


understand  •  2  steps  to  understand  the  Context-Independent 
the  spec’s  specifications  of  the  of  the  materiel  system-of-systems. 


•  1  additional  step  to  understand  the  mission  and  system 
specifications  linkage  (retain  Context-Dependence  link). 


CD 


LD 

co 

CD 


design  the  •  SE  steps  to  design  the  system  to  the  specifications  given 
System _ the  Mission,  Task,  HD  context  understanding. _ 


determine  •  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
THE  RESULTS  data. 


REPORT 
THE  RESULTS 


1  step  to  format  and  report  the  results. 
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RDECOM 


If  Human  Dimension  (HD)  is  the 

“System” 


The  V  S.  Army  Concept  for  the 

\  f -j 

Human 
Dimension 


Social  Behavior  (9) 

•  Affects,  Emotions,  and  Moods 

•  Cultural  Awareness  (CA)  &  CA  Training 

•  Ethics  &  Values  /  Morals  &  Beliefs 

•  Group  Dynamics  /  Group  Interactions 

•  Interpersonal  Relations 

•  Leadership  &  Leadership  Training 

•  Networking 

•  Personnel  Issues  /  Recruitment  &  Retention 

•  Quality  of  Life 


Human  Dimension  is  parsed  by 
three  behavioral  domains:  Social, 
Cognitive,  and  Physical 


Cognitive  Behavior  (12) 

Attention  &  Memory 

Cognitive  Workload 

Comprehension  /  Understanding 

Creativity  &  Imagination 

Decision  Making 

Learning 

Motivation 

Pattern  Recognition 

Perception 

Problem  Solving 

Projection  &  Planning 

Situation  Awareness 


Physical  Behavior  (13) 

Anthropometry 

Biological/Physiological  Mechanisms 

Biomechanics 

Endurance  &  Tolerance 

Fitness  &  Strength 

Health  Protection  /  Preventive  Medicine 
Medical  Intervention 
Mobility  &  Dexterity  /  Movement 
Nutrition 

Physical  Adaptability  /  Survivability 

Physical  Comfort 

Sensing 

Task  Execution  /Action  &  Reaction 


MANPRINT  Domains 


MANPRINT 

Domains 


HFE  Taxonomy* 


1.  Manpower 

2.  Personnel 

3.  Training 

4.  Human  Factors 
Engineering 

5.  System  Safety 

6.  Health  Hazards 

7.Soldier 

Survivability 


Environment 


Information/ 

Communications 


Characteristics/ 

Organization/ 

Design 

Workspace 

Design 


Methods  & 
Techniques 


*  Human  Factors  Engineering  (HFE)  Taxonomy 
adapted  from  Salvendy  (2006) 
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Current:  Warfighter  to  Human 
Dimension  (and  back) 


MANPRINT  Domains 


1.  Manpower 

2.  Personnel 

3.  Training 

4.  Human 
Factors 
Engineering 


5.  System 
Safety 

6.  Health 
Hazards 

7.  Soldier 
Survivability 


LVC  Interaction 

Distributed 

Behavioral 

Design 

Simulation 

Representation 

1 

Methods 

LJ  l 


Training 
Methods  & 
Techniques 


Training 

Research 

Domains 


Tracking  &  Experimental 

Locomotion  Methods 


Intelligent/  ■  Rapid  ■  Model  &  ■  Embedded 

Autonomou  H  Development  I  Simulation  H  Training 
Tools  I  Development 


Source  Data  ^  Objectives, 
Standards 
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Warfighting: 

f(personnel, 

materiel, 

mission) 


Optional  Shorted 


Current  ARCIC 
Function 


=  Integration  Process 


Requirements  Flow 


Solutions  Delivery 


wuMSm 

ARCIC  HD 
Taxonomy 


!> 


Current:  Warfighter  to  Human 
Dimension  (and  back) 


Training  Taxonomy 


jo 

co 


Tri-Service 

IMPRINT 

Model 

Mapping 


Tri-Service 

MANPRINT 

Taxonomy 


MANPRINT  Domains 


1.  Manpower 

2.  Personnel 

3.  Training 

4.  Human 
Factors 
Engineering 


5.  System 
Safety 

6.  Health 
Hazards 


7.  Soldier 
Survivability 


=  Integration  Process 


Human;  Facte 
Engineering  Taxonomy 


Stejnd  Shorted 


Current  ARCIC 
Function 


<z 


Requirements  Flow 


Solutions  Delivery 
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Alternate:  Warfighter  to  Human 
Dimension  (and  back) 


=  Integration  Process 


MANPRINT  Domains 


1.  Manpower 

2.  Personnel 

3.  Training 

4.  Human 
Factors 
Engineering 


5.  System 
Safety 


Proposed  ARCIC 
Linkage:  Based  on 
the  OSD/P&R 
Assessment  Process 


MBT&E  Framework  —  v2 


Process/Products 


Commander’s  Task 
to  Subordinates 


- 1 - 

Transition  to  Allocating  Mission  Means 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


Commander’s  Task 
to  Subordinates 


Transition  to  Allocating  SoS  Means 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


System  Attributes 


Transition  to  Allocating  Materiel  Means 

± 


Systems  Engineering 

•  Functional  Baseline 

•  Allocated  Baseline 

•  Product  Baseline 


fAMSAA  | 

Excellence  in  Analysis 


Capability  =  Set  of  Tasks  +  Desired  Result 


Desired 
End  State 


Enables 


Operations  (Mission  Tasks) 

•  UJTLs 

•  Service  TLs 

•  Implied  Tasks 


Desired  Mission 
Task  Results 


Mission  Task  Capability 


System-of-Systems  Tasks 
Service  TLs 
Implied  Tasks 
Collective/Individual  Tasks 


Desired  SoS 
Task  Results 


T 


SoS  Task  Capability 


System  Performance 
•Functions  (shall  do) 

•  “shall  be’s” 

System  Performance 


Desired  System 
Performance  Results 


Essence: 
What  must  be 
accomplished 
and  Why 


Implementation: 
DOTMLFP 
How  and  Who 


DOTMPF 

Component 


Benefits 


Mission-base  SE  in  an  SoS  Environment  enables 
identification,  trade-off,  and  design  of  System 
structure  and  allocation,  characteristics  and 
performance  to: 

-  Prioritize  by  stressor  relevance  to  End-Results, 
Mission-Task  effectiveness,  SoS-Task  performance, 
Human  Dimension,  and  Operational  Variables  as  well 
as  stressor  relevance  to  traditional  Materiel  and 
Technology  considerations. 

-  Articulate  the  impact  of  System  capabilities  in  the 
language  of  the  warfighter  as  expressed  in  the 
originating  JCIDS  FAA,  FNA,  and  FSA 


FAA 

FNA 

FSA 


Functional  Area  Analysis 
Functional  Needs  Analysis 
Functional  Solution  Analysis 


RDECOM 


Summary 


•  For  identified  gaps  in  required  Capability,  employ  a  Mission-Essential-Task- 
List  (METL,  with  measure,  conditions  and  standards)  centric  Systems-of- 
Systems  (SoS)  approach  to  complete  the  (traditional)  materiel  centric 
Systems  Engineering  “V”  (SE-V)  in  the  conception,  development,  evaluation, 
and  fielding  of  DOTMLPF  (Doctrine,  Organization,  Training,  Materiel, 
Leadership,  Personnel,  and  Facilities)  solutions  for  the  warfighter 

•  The  key  is  systematically  deriving,  retaining,  and  employing  Mission,  Task, 
and  Human  Dimension  context  throughout  the  extended  SE-V  in  the  SoS 
environment  to  need  analysis,  trade  study,  design  allocation,  and  capability 
assessment  results  in  traditional  materiel  centric  terms  and  their 
relationships  and  contributions  both  direct  and  indirectly  to  the  impact  on 
warfighting  operational  performance  and  mission  effectiveness. 

•  This  approach  is  an  application  of  the  Missions  and  Means  Framework 
(MMF)  that  is  tailored  to  be  compatible  with  the  existing  Army  Guidance  for 
Employment  of  Forces  (GEF)  directive  for  certifying  operational  forces  prior 
to  deployment,  the  recently  released  USD/AT&L  Systems  Engineering 
Guide  for  a  SoS  Environment,  and  the  emerging  ATEC  Mission-Base  Test  & 
Evaluation  (MBT&E)  and  DOT&E  Joint  Test  &  Evaluation  Methodlodgy 
(JTEM)  a  frameworks  and  procedures.  . 


RDECOM 


Backup 


RDECOM 


Definitions 


(lof  2) 


•  System  (S)*  -  A  functionally,  physically,  and/or  behaviorally  related  group  of  regularly 
interacting  or  interdependent  elements;  that  group  of  elements  forming  a  unified 
whole  [JP  1-02  &  JP  3-0] 

•  Capability*  -  is  the  ability  to  achieve  a  desired  effect  under  specified  standards  and 
conditions  through  combinations  of  ways  and  means  to  perform  a  set  of  tasks  [CJCS, 
2007(2)]. 

•  Family  of  Systems  (FoS)*  -  a  set  of  systems  that  provide  similar  capabilities  through 
different  approaches  to  achieve  similar  or  complementary  effects  [CJCS,  2007(1)]. 

•  System  of  Systems  (SoS)*  -  is  defined  as  a  set  or  arrangement  of  systems  that 
results  when  independent  and  useful  systems  are  integrated  into  a  larger  system  that 
delivers  unique  capabilities  [DoD,  2004(1)]. 

•  Both  individual  systems  and  SoS  conform  to  the  accepted  definition  of  a  system  in 
that  each  consists  of  parts,  relationships,  and  a  whole  that  is  greater  than  the  sum  of 
the  parts;  however,  although  an  SoS  is  a  system,  not  all  systems  are  SoS.  * 


Taken  from  the  DoD  SE  Guide  for  SoS 
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Definitions  <2  of  2) 


•  Virtual  SoS*  -  Group  lacks  a  central  management  authority  and  a  centrally  agreed 
upon  purpose  for  the  system-of-systems. 

•  Collaborative  SoS*  -  Group  component  systems  interact  more  or  less  voluntarily  to 
fulfill  agreed  upon  central  purposes. 

•  Acknowledged  SoS*  -  Group  has  recognized  objectives,  a  designated  manager, 
and  resources  for  the  SoS;  however,  the  constituent  systems  retain  their  independent 
ownership,  objectives,  funding,  and  development  and  sustainment  approaches. 
Changes  in  the  systems  are  based  on  collaboration  between  the  SoS  and  the 
systems. 

•  Directed  SoS*  -  The  group,  an  integrated  system-of-systems,  is  built  and  managed 
to  fulfill  specific  purposes.  It  is  centrally  managed  during  long-term  operation  to 
continue  to  fulfill  those  purposes  as  well  as  any  new  ones  the  system  owners  might 
wish  to  address.  The  component  systems  maintain  an  ability  to  operate  independently, 
but  their  normal  operational  mode  is  subordinated  to  the  central  managed  purpose. 

•  When  a  group  of  systems  morphs  into  an  SoS  -  synergy  occurs  between  the  various 
systems 

•  When  a  group  of  systems  is  not  an  SoS  -  synergy  between  the  various  elements  does 
not  occur 


Taken  from  the  DoD  SE  Guide  for  SoS 
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Human  Factors  Engineering 

Taxonomy 


Training 


LVC  Interaction 
Design 


Sensory 

Interfaces 


Tracking  & 
Locomotion 


Synthetic 

Environments 


Rapid 

Development 

Tools 

Source  Data 
Processing 


Real-time 

Dynamic 

Physics- 

based 

Environments 


Training 

Research 

Domains 

I 


Embedded 

Training 


Medical 

Training 


Immersive 

Learning 
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SoS  Elements  Crossed  with 
Context  Dependency 


Mission 

Doctrine 

Organization 

Organization 

Success 

Measures 

Context 

Dependent 

Major  Combat 
Ops 

Match 

Match 

Exist 

Context 

Dependent 

Stability  & 
Support  Ops 

Mismatch 

Mismatch 

Not  well  developed 

Context 

Independent 

Maintain  Force 

Match 

Match 

Exist 

Context 

Independent 

Establish 

Cordon 

Match 

Match 

Exist 

RDECOM 


Forms  Crossed  with  Acquisition  / 
Organizational  Processes  Wmm 


System 

Directed 

SoS 

Acknowledged 

SoS 

Collaborative 

SoS 

Virtual 

SoS 

Governance 

Feudal 

Central  with 
limited  local 
autonomy 

(France, 

Russia) 

Federation  with 
states  rights 
freedom  of  action 

(US,  Canada) 

Tribal 

Fair  market 
economy 

Conflict 

Resolution 

Adjudicated 

Adjudicated 

Negotiated 

Competed 

Pair-wise 

consent 

Schedule 

Synchronized 

Synchronized 

Emergent 

Synchronized 

Asynchronous 
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Forms  Crossed  with  Data  / 
Manaqement 


System 

Directed 

SoS 

Acknowledged 

SoS 

Collaborative 

SoS 

Virtual 

SoS 

Enterprise 

No 

Yes 

Yes 

No 

No 

Communities 
of  interest 

Yes, 

A  Priori  by 
design, 

Stable  over 

whole 

period 

Yes, 

A  Priori  by 

design, 

evolving, 

stable  when 

eventually 

complete 

Yes, 

Evolving  during 
development,  then 
stable  when 
complete 

Yes, 

Evolving  during 
development, 
then  stable 
when  complete 

Yes, 

Morphs  as  the 
partners  change 

local 

Yes 

Abrams 

Yes 

SoSCOE 

Yes 

USMTFA/MF 

Yes 

AKO 

Yes 

Proprietary 

Protocols 

SoS  Forms  Crossed  with 
Acquisition  Life  Cycle  Management 


System 

Directed 

Acknowledged 

Collaborative 

Virtual 

SoS 

SoS 

SoS 

SoS 

Program 

Risk 

Internal 

Internal  & 
external 

Internal  &  external 

Internal  &  external 

internal 

AoA- 

TRADOC 

Early,  simple,  and 
complete 

Extensive  and 
on-going 

Extensive  and  on¬ 
going 

Extensive  at  S 
level  but  not  at 

SoS  level 

Partner  selection  in 
business  sense 

Rqmts  -  pm 

Directed 

Directed 

Directed 

Negotiated 

Pair-wise  Consent 

Trade 
Studies  - 

PM 

Early,  simple,  and 
complete 

Extensive  and 
on-going 

Extensive  and  on¬ 
going 

Extensive  at  S 
level  but  not  at 

SoS  level 

Partner  selection  in 
business  sense 

Build  -  pm 

Well  defined 
prime 

Multiple  primes 
&  PMs 

Multiple  primes  & 

PMs 

Competitive 

By  independent 
partners 

Integrate, 
V&V  -  PM 

Internal  to  defined 
prime 

Over  multiple 
primes  &  PMs 

Over  multiple  primes 
&  PMs 

Over  multiple 
primes  &  PMs 

Internal  to 
partnership 

T&E  -ATEC 

Independent 

oversight 

Independent 

oversight 

Independent 

oversight 

By  developers 

By  customers 
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Mission-Based  T&L 
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US  Army  Evaluation  Center 
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MBT&E  Background 


MBT&E  Framework 
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RDEcnM*  Compatibility 


•  ATEC  Mission  Based  Test  &  Evaluation  is  consistent  with 

-  OSD  P&R  directives  for  reporting  METL-based  Readiness 

-  Joint  GEF  certification  for  Deployment  of  Operational  Forces 

-  JCIDS  Capability  based  Acquisition 

-  DoD  Systems  Engineering  guide  for  Systems  of  Systems  (SE  for  SoS) 

-  DOT&E  Joint  Test  &  Evaluation  Methodology  (JTEM) 


C  Based  on  Generic,  General  Purpose 
Measures/Conditions/Standards 

-  TPFData  List  TO&E  (standing)  C-METL 

-  TPFDeploment  Data  MTOF  (transient)  D-METL 


Based  on  Mission,  Situation  Specific 
Measures/Conditions/Standards 


OSD  P&R  Readiness 
Joint  GEF  Certification 


TPFDL:  Time  Phased  Force  Data  List  for  deployment  planning 
TPFDD:  Time  Phased  Force  Deployment  Data  for  deployment  execution 
TO&E:  Table  of  Organization  and  Equipment  for  a  standing  unit 

MTOF:  Mission  Task  Organized  Force  (modeling  &  simulation  term) 

C-METL:  Core  METL  defined  by  Army  Force  Generation  (ARFORGEN)  for  commonality  between  units  of  the  same  type 
D-METL:  Directed  METL  defined  by  Army  Force  Generation  (ARFORGEN)  for  a  unit  with  specific  deployment  orders 
GEL:  Joint  Guidance  for  Employment  of  Forces  for  unit  certification  prior  to  deployment 

P&R:  OSD  Personnel  &  Readiness  directives  for  METL-based  readiness  reporting 

JCIDS:  Joint  Capability  Integrated  Development  System 


MBT&E  Framework 
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Case  Study  Introduction 
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Case  Study 

•The  purpose  of  this  case  study  is  to  to 
present  the  concepts  and  operation  of  the 
MBT&E  methodology  using  a  example. 

•The  intent  is  to  involve  the  audience  in  the 
development  of  an  MBT&E  strategy  and  to 
focus  discussions. 

•The  information  presented  in  the  case 
study  is  fictional. 
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Case  Study 

•  Joint  Capabilities  Integration  and  Development  System 
conducting  a  capabilities-based  assessment  of  modular 
combat  brigades. 

•  Brigade  Combat  Team  supported  by  Reconnaissance 
Attack  battalion  from  Combat  Aviation  Brigade. 

•  Functional  Area  Analysis 

-  Combat  brigades  required  to  support  noncontiguous  operations. 

•  Ground  units  conducting  simultaneous  full  spectrum  operations  in  separate  locations. 

•  Aviation  units  providing  support  to  simultaneous  operations  (one  aviation  team 
supporting  more  than  one  ground  unit  operation). 

-  Capability:  Attack  time-sensitive  targets  based  on  maneuver  ground  units  call 
for  fire  (eyes-on)  and  limited  intelligence  (developing  situation). 
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Case  Study 


Function  Need  Analysis 

-  Gap  1:  Time  sensitive  targets  need  to  be  engaged  within  15  minutes. 

•  High  priority  targets,  once  identified,  need  to  be  destroyed  before  they  have  a  chance 
to  escape  or  hide  in  dense  urban  terrain,  approx  15-20  minutes. 

•  Current  aviation  unit  response  time  from  call  for  fires  to  prosecuted  target  as  much  as 
45  minutes,  depending  on  current  location  and  location  of  the  support  call  (50  nm). 

•  Current  quick-response  call  for  fires  from  other  assets  (artillery  and  current  air- 
launched  munition  set)  not  reliably  available  or  not  desired  due  to  need  to  minimize 
collateral  damage  in  dense  urban  terrain. 

-  Gap  2:  Immediate  response  (<15  minutes)  and  extended  surveillance  (>45 
minutes)  needed  to  develop  situational  intelligence. 

•  Success  seen  in  using  pre-planned  reconnaissance/surveillance  assets  (RSAs)  to 
observe  suspicious  behavior  of  initial  target  leading  to  other,  more  valuable,  targets. 

•  Currently,  initial  targets  are  being  lost  due  to  response  time  from  observation  to  re¬ 
tasking  of  RSAs  and  inability  of  ground  units  to  continue  to  surveil  initial  targets  in 
dense  urban  terrain  unobserved.  Most  targets  lost  within  15  minutes. 

•  Currently,  targets  taking  as  much  as  45  minutes  to  move  from  initial  observation  area 
to  centralized  base.  (Based  on  pre-planned  RSA  missions.) 


Case  Study 

•  Functional  Solution  Analysis 

-  Reconnaissance/Attack  System  (RAS) 

•  Air-launched  loitering  sensor/munition. 

•  Man-in-the-loop  control  and  targeting  after  launch. 

•  IR  and  SAL  seeker 

•  RAS  ICD/Draft  CDD 

-  Air-launched  (AH-64D,  F/A-18E/F,  and  UAS  based  on  aircraft 
supporting  ground  operations). 

-  Loiter  Capability  (>45  minutes,  based  on  time  it  takes  aviation  units  to 
move  from  one  location  to  another.) 

-  Multi-purpose  warhead  (Structure,  Vehicle,  Personnel  targets,  based  on 
expanded  target  set.) 

-  Range  (50nm,  based  on  distributed  operations.) 

-  Time  to  Target  (<1 5  minutes  to  50  nm) 

-  Probability  of  single-shot  kill  (Pssk)  (>80%) 
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MBT&E  Procedure 
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MBT&E  Procedure 

Step  1 

Collect  Information 
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Collect  Information  (Step  1) 


Currently 

building/maintaining 
document  list. 


Document  Title 

Who  to  ask? 

Functional  Area  Analysis 

TRADOC 

Functional  Need  Analysis 

TRADOC 

Functional  Solution  Analysis 

TRADOC 

Analysis  of  Alternatives 

TRADOC 

Operational  Mode  Summary/Mission  Profile 

TRADOC 

Initial  Capabilities  Document 

DoDAF  OV-1  (Appendix  A) 

TRADOC 

Capabilities  Documents  (CDD,  CPD) 

DoDAF  OV-1  (Appendix  A) 

DoDAF  OV-2  (Appendix  A) 

DoDAF  OV-4  (Appendix  A) 

DoDAF  OV-5  (Appendix  A) 

DoDAF  OV-6C  (Appendix  A) 

DoDAF  SV-2  (Appendix  A) 

DoDAF  SV-4  (Appendix  A) 
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Functional  Area  Analysis 
Functional  Need  Analysis 
Functional  Solution  Analysis 
Analysis  of  Alternatives 
Operational  Mode  Summary/Mission  Profile 
Initial  Capabilities  Document 
DoDAF  OV-1  (Appendix  A) 

Capabilities  Documents 

DoDAF  OV-1  (Appendix  A) 

DoDAF  OV-2  (Appendix  A) 

DoDAF  OV-4  (Appendix  A) 

DoDAF  OV-5  (Appendix  A) 

DoDAF  OV-6C  (Appendix  A) 

DoDAF  SV-2  (Appendix  A) 

DoDAF  SV-4  (Appendix  A) 

DoDAF  SV-5  (Appendix  A) 

DoDAF  SV-6  (Appendix  A) 

COI/Cs 

Army  Universal  Task  List,  FM  7-15 
Unit  Mission  Training  Plan 
Support  Unit  Mission  Training  Plans 
Universal  Joint  Task  List,  CJCSM  3500 
System  Threat  Assessment  Report 
Technology  Development  Strategy 
Acquisition  Strategy 

Work  Breakdown  Structure 
Materiel  Fielding  Plan 
System  Performance  Specification 


Who  to  ask? 


TRADOC/School  Flouse 
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Joint  Staff  (online) 


f  I 

=>  £ 


1  1 


i  I  i  I 

s  i 


I  1 

5  1 
1  1 


MBT&E  Procedure 
Steps  2-5 

Understand  the  Mission 
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MBT&E  Procedure 

Step  2 

Define  the  Mission 

Context 
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Focused  Mission  Case 

•  Strategic  End  State:  Restore  secure  stable  environment 

-  Mission  1 :  Restore  basic  services 

•  [Basic  services  restored  to  80%  of  the  population.] 

-  Mission  2:  Create  a  professional  security  forces 

•  [Security  forces  able  to  maintain  order.] 

-  Mission  3:  Provide  area  security. 

•  [Create  conditions  for  government  to  function  and  citizens  to  live,  work  and  play.] 


•  Scenario  for  Mission  3:  BCT  establishes  platoon  sized  Combat 
Outposts  in  cleared  areas  to  prevent  local  insurgent  forces/foreign 
fighter  cells  from  operating.  Units  conduct  patrolling  and  presence 
operations  to  reassure  civilian  population  and  disrupt  insurgent 
operations  in  the  area  of  operations.  Attack  Recon  Company  (AH  64) 
supports  ground  units  with  aerial  recon,  cooperative  engagement  of 
High  Payoff  Targets  and  attack  of  time  sensitive  targets. 
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Case  Study 
(Fictional) 

Scenario  Operational  Setting 


•  Situation 

-  Brigade  Combat  Team  conducting  full  spectrum  operations  within  an  assigned  Area  of 
Operations. 

•  Active  insurgency  with  mix  of  discontented  local  tribes  and  foreign  extremist  fighters  equipped 
with  modern  communications  and  MANPADS. 

•  Subordinate  Combined  Arms  Battalions  operating  platoon  sized  Combat  Outposts  throughout 
the  AO 

-  Insurgent  leadership  conducts  planning  and  coordination  meetings  at  varying  times 
and  locations  to  reduce  vulnerability  to  attack  with  meeting  times  normally  limited  to 
less  than  1  hour. 

-  Insurgent  gatherings  are  COIR  for  BCT  Cdr  and  insurgent  leaders  are  High  Payoff 
Tgts 

•  BCT  Mission:  Secure  the  BCT  AO  in  order  to  set  conditions  for  local 
civilians  to  carry  on  normal  activities  and  handover  responsibility  to  HN 
Forces 

•  Execution 

-  Commander’s  Intent 

•  Locating  and  eliminating  insurgent  leadership  is  critical  to  our  ability  to  establish  a  secure 
environment  for  the  civilian  population. 

•  Equally  important  is  the  need  to  ensure  positive  identification  and  avoid  collateral  damage. 
Unintentional  civilian  casualties  will  severely  undermine  the  mission  by  throwing  support  to  the 
insurgents. 
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Task  Organization 


Brigade  Combat  Team  (BCT)  task 
organized  with  3  Stryker  battalions 
and  1  Attack  Helicopter  battalion 
from  the  Corps  Aviation  Brigade 
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Current  Situation 


Combat 

outpost 


Suspect 


Meeting 


Site 


Combat  Outpost  receives  a  tip  indicating  a  likely 
meeting  of  local  insurgent  leadership  group. 
Informant  provides  an  address  and  warns  the 
insurgents  have  OP’s  on  access  routes  to  warn 
of  approaching  vehicles.  Security  is  posted  in 
civilian  trucks,  armed  with  MAN  PADS.  R/A  Co. 
supporting  area  security  performing  area  recon 
and  attack  by  fire  tasks  to  locate  and  destroy 
HPTs 
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AO 

DAGGER 


Current  Situation 


Combat 

outpost 


Suspect 


Meeting 


Site 


AO  'u  ~  j 1 

DAGGER 


Combat  Outpost  reports  information.  BCT  Cdr 
wants  immediate  “eyes  on”.  AH-64  platoon  from 
support  A/R  company  uses  complex  terrain  to 
south  to  observe  NAIs  along  routes  to  suspected 
meeting  location.  AH  64  launches  RAS  to 
maintain  surveillance  as  suspicious  vehicles 
begin  arriving  and  to  reducerTarget  engagement 
window.  Imagery  of  suspected  insurgents 
gathering  is  transmitted  to  AH-64  and  to  CGS  in 
BCT  TOC.  BCT  BfSB  employs  other  intel 
disciplines  to  confirm  insurgent  leadership 


gathering. 
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NAI  1 


1.  Insurgent  meeting  begins  to  break  up: 

a) No  joint  aircraft  immediately  available 

b) Joint  attack  aircraft  on  station  and  available  to 
respond  in  less  than  5  min 

2a.  AH  64  cleared  to  attack  with  RAS 

2b.  Strike  request  to  joint  air.  RAS  imagery 

shared. 


■■  •&& 


■  j.  1 1 


Operational  Conditions 

•  Mission  3:  In  RAS  context. 

-  Conduct  Dynamic  Targeting  (OP  3.1.9)!  To  achieve  timely  and  accurate 
detection  and  prosecution  of  time-sensitive  targets  through  integration  of  dynamic  ISR 
support  and  operations  in  support  of  the  operational  level  JFC's  intent.1 

-  Conduct  Air  to  Surface  Attack  (ART  3.3.1 .2):  Use  fixed-  and  rotary-wing 
aircraft-mounted  weapon  systems  to  destroy,  suppress,  or  neutralize 

equipment  (including  aircraft  on  the  ground),  materiel,  personnel,  fortifications,  and 
facilities.2 

•  Summary  of  operational  conditions  (METT-TC) 

-  Mission:  Employ  Lethal  Fire  Support  (ART  3.3.1) 

•  Joint  Task:  TA  3.2.1  Conduct  Fire  Support 

•  Army  Task:  ART  1 .4.1  Conduct  Lethal  Direct  Fire  against  a  Surface  Target 

-  Enemy:  Vehicles,  and  Personel  in  open.  Personnel  in  buildings. 

-  Troops:  SBCT;  R/A  Company  (AH  64);  joint  air  attack  teams  (USAF  and  USN). 

-  Terrain  &  Weather:  Urban,  desert,  mountain,  all  weather 

-  Time  Available: 

•  Clock  Hours:  Daylight,  Night  and  Crossover. 

•  Planning  Time:  Extensive  coordination  planning.  Engagement  plan  developed 
as  situation  unfolds. 

-  Civil  Considerations:  Limited  collatoral  damage.  Positive  ID  of  targets  necessary 
(friendlies  in  the  area). 


1 .  CJCSM  3500. 04C  Universal  Joint  Task  List 

2.  FM  7-15  The  Army  Universal  Task  List 
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MBT&E  Procedure 

Step  3 

Develop  the  Mission 

Tasks 
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Mission  Analysis 

•  Area  Reconnaissance  and  Lethal  Direct  Fire  against 
surface  targets  (As  requested  by  supported  BCT  Cdr). 

-  Plan  Mission  [Mission  planned  and  mission  data  loaded  on  data 
cartridge.] 

•  Weapon  Load  Planning. 

•  Complete  Performance  Planning  Calculations. 

-  Prepare  for  Mission  [Aircraft  report  lgo’  status  and  lined  up  at 
tactical  assembly  area.] 

•  Load  Aircraft  and  Weapon  Status  Check. 

•  Lineup  for  takeoff. 

-  Execute  Mission  [R  and  A  mission  executed  as  requested  by 
ground  units.] 

•  TAAtoAO 

•  Support  Ground  Unit  in  AO 

•  AOtoTAA 


24  bb 


Mission  Analysis 

•  Support  Ground  Units  in  AO 

-  Check  in  with  ground  units  [Contact  with  ground  units  is  established] 

•  Accept  attack  mission  [Attack  mission  is  accepted  by  aviation  unit.] 

•  Employ  RAS  [Aviation  units  arrive  at  engagement  area  (EA).] 

-  Launch  RAS  Munition  [RAS  munition  is  launched  and  is  flying  normally.] 

-  Guide  RAS  to  EA  [RAS  munition  arrives  in  target  area.] 

-  Gather  situational  information  [SA  is  understood  and  target  is  identified.] 

•  Decide  on  employment  technique  [Engagement  technique  is  selected.] 

•  Engage  Target  [Target  is  engaged  and  destroyed.] 

-  1 .  Engage  with  RAS  [Target  is  destroyed  by  RAS.] 

-  2.  Engage  with  onboard  munitions  [Target  is  destroyed  by  selected  munition.] 

-  3.  Call  in  Joint  Air  Attack  Team  [Target  is  destroyed  by  supporting  aircraft.] 

•  Battle  Damage  Assessment  [Target  state  is  determined.] 

•  Decide  on  re-attack  or  return  to  supporting  position  [Followon  action  is 
identified.] 
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Mission  Analysis 


•  Support  in  AO  - ART  3. 3. 1.2  Conduct  Air  to  Surface 

Attack 


-  Check  in  with  ground  units  — » 

•  Accept  attack  mission - > 

•  Employ  RAS  - » 

-  Launch  RAS  Munition - ► 

-  Guide  RAS  to  AO  - -> 

-  Gather  situational  information  — > 

•  Decide  on  employment 

technique  * 

•  Engage  Target  - > 

-  1 .  Engage  with  RAS - > 

-  2.  Engage  with  onboard 

munitions  * 


-3.  Call  in  JAAT 


Battle  Damage  Assessment  ■ 

Decide  on  re-attack  or  return 
to  supporting  position 


ART  5. 1.2.1  Est.  Coordination  &  Liaison 
ART  5. 1.1.1  Receive  the  Mission 
ARTEP  01-2-5183  Tactical  Air  Movement 

RAS  7  Launch  RAS 
RAS  1 5  Control  RAS 

ART  2.3.4  Conduct  Surveillance;  ART5.1 .4.1 
Monitor  Situation 

TC  251-1422  Perform  Firing  Techniques 
same  as  above 

RAS  10  Modify  RAS  Warhead  Setting;  RAS  9 
Modify  RAS  Employment  Setting 

TC  251-1458  Engage  with  Point  Target  Weapon 
System 

ARTEP  01-2-0106.01  Conduct  JAAT  Operations 
ART  5. 1.4. 3.1  Conduct  BDA 
ART  5. 1.4. 3. 3  Provide  re-attack  recommendation 
TC  251-1405  Transmit  Tactical  Report 
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MBT&E  Procedure 

Step  4 

Develop  the  Supporting 

Tasks 
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SupportingT  asks 

•  Conditional  Tasks 

-  Deny  Enemy  Engagement  [Enemy  can  not  engage  aircraft.] 

•  ART  5.7.1  Protect  against  Enemy  Hazards  in  the  A.O. 

-  Jettison  Launcher  [Launcher  with  munitions  is  jettisoned.] 

•  TC  251-1070  Respond  to  Emergencies 

•  Enabling  Tasks 

-  Replace  IR  Coolant  Bottle  [Spent  coolant  bottle  is  removed, 
replacement  bottle  is  installed,  and  missile  provides  a  “go” 
status.] 

•  ARTEP  01-2-5212  Perform  helicopter  repairs  and  required  inspections  of 
aircraft  subsystems 

-  Training  [Aircrew  is  trained  and  certified  in  RAS  control  tasks.] 

•  ART  5.1 .2.5  Conduct  Pre  Operations  Checks  and  Inspections 
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MBT&E  Procedure 

Step  5 

Identify  Task  Capabilities 
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Associate  Tasks  with  Requirements 


-  Air  to  Surface  Attack  (ART  3.3.1  Employ  Lethal  Fire  Support)  {Time  to 
target  observation<15  minutes.} 

-  Support  SBCT  in  AO  (ART  1.4.1  Conduct  Lethal  Direct  Fire) 

-  Employ  RAS  (RAS  7)  {Positive  control  range  >50  nm.} 


Engage  with  RAS  (TC  251-1522  Perform  Firing  Techniques) 
{Probability  of  Single-Shot  Kill  (Pssk)  >80%.} 


Means 

Aircraft,  RAS  munition, 
communication  equipment. 


Ways 

Close  air  support  tactics, 
techniques  and  procedures; 
Aircrew  training  manuals;  unit 
tactical  operations  procedures. 


Destroy  or  disarm  existing 
paramilitary  forces 


Enables 


Engage  with 
RAS 


Capability 


Target  is 
Destroyed 


METT-TC 

From  Mission 
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Afternoon  Break 
1500-1515 
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MBT&E  Procedure 

Steps  6-8 

Understand  the  System  & 
Associate  Capabilities  with  Attributes 
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MBT&E  Procedure 

Step  6 

Determine  the  SoS 
Components 
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SoS  Description 

•  AH-64D  [Transport  /  deliver  missile.] 

-  Launcher  [Control,  communicate,  launch  missile.] 

-  Tactical  Data  Link  [Control  RAS  during  flight.] 

-  Avionics  [Communicate  with  ground  forces.] 

•  RAS  Munition  [1.  Provide  situational  information,  2.  Destroy  target.] 

•  Simulator  [Exercise  aircrews  in  RAS  TTPs.] 

•  Remote  Designator  [Designate  Target] 

•  Mission  Planning  System  [Load  and  performance  planning 
dowloaded  on  cartridge.] 
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SoS  Description 

•  Aligned  with  PM’s  Work  Breakdown  Structure 

•  RAS  Munition  [1.  Provide  situational  information,  2.  Destroy 
target.] 

-  Seeker  [Provide  situational  images,  acquire  and  track  target.] 

-  Motor  [Provide  Thrust.] 

-  Warhead  [Provide  lethal  effects.] 

-  Guidence  and  Control  System  [Guide  munition.] 

-  Wings  [Provide  lift.] 

-  Tactical  Data  Link  (TDL)  [Communicate  with  controlling  asset.] 
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SoS  Description 

•  Aligned  with  Critical  Technologies  List 

•  Motor  identified  as  a  critical  technology 

-  Proposed  miniture  air-breathing  engine  not  previously 
demonstrated. 

-  Planned  wind  tunnel  test  prior  to  MS  B. 

•  Miniature  control  data  link. 

-  Capability  to  provide  positive/fail-safe  control  of  a  miniature 
munition  not  previously  demonstrated. 

-  Planned  captive  flight  testing  prior  to  MS  B. 
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MBT&E  Procedure 

Step  7 

Develop  System 
Attributes 
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Associate  Materiel  with 
Requirements 

•  RAS  Munition  [1.  Provide  situational  information,  2. 
Destroy  target.]  {In-flight  Reliability;  %  non-essental 
function  failure  >  93%.}  {Loiter  time  >  45  minutes.} 

-  Seeker  [Provide  situational  images,  acquire  and  track  target.] 

{Minimum  Delta-Temperature.}  {Operate  with  all  semi-active 
laser  code  frequencies.} 

-  Motor  [Provide  Thrust.]  {Low  observable  smoke.} 

-  Warhead  [Provide  lethal  effects.]  {Probability  of  a  kill  given  a 
hit;  Pk/h,  >95%.} 

-  C&G  [Guide  munition.]  {Probability  of  a  hit  given  a  shot;  Ph/s, 
>90%.} 

-  Tactical  Data  Link  [Receive  control  signals.  Transmit  target 
information.]  {Positive  communication  link  range  >60  nm.} 
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MBT&E  Procedure 

Step  8 

Associate  Capabilities 

with  Attributes 
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Linking  task  to  materiel 

-  Air  to  Surface  Attack  (ART  3.3.1  Employ  Lethal  Fire  Support)  {Time  to 
target  observation  <15  minutes.} 

--  Support  Ground  Unit  in  Aos  (ART  1.4.1  Conduct  Lethal  Direct  Fires) 

- —  Employ  RAS  (RAS  7)  {Positive  control  range  >50  nm.} 

- Engage  with  RAS  (TC  251-1522  Perform  Firing  Techniques) 

{Pssk  >80%.} 

•  Aircraft  TDL  [Control  RAS  during  flight.]  {Positive  communication  link 
range  >  60nm} 

•  RAS  Munition  [1.  Provide  situational  information,  2.  Destroy  target.]  {Prel 
%  non-essental  function  failure  >  93%.}  {Loiter  time  >  45  minutes.} 

•  Seeker  [Provide  situational  images,  acquire  and  track  target.]  {Minimum 
Delta-Temperature.}  {Operate  with  all  semi-active  laser  code  frequencies.} 

•  Warhead  [Provide  lethal  effects.]  {Pk/h,  >95%.} 

•  C&G  [Guide  munition.]  {Ph/s,  >90%.} 
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Enabling  Attributes 

•  Materiel  system  attributes  that  apply  to  all  tasks. 

-  Electromagnetic  Survivability  [Materiel  system  is 
protected  from  electromagnetic  interfearance.] 

-  Information  Assurance  [Materiel  system  is  protected 
from  10  attack  and  critical  information  is  not 
compromised.] 

-  Reliability  [Materiel  system  has  sufficient  time 
between  failures  to  enable  execution  of  soldier  tasks.] 

-  Maintainability  [Materiel  system  is  maintainable  fy 
field/fleet  maintainers  and  operators.] 
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MBT&E  Procedure 
Steps  9-15 
Design  the  T&E 
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MBT&E  Procedure 

Step  9 

Unconstrained 
Operational  Conditions 
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Operational  Conditions 

-  Engage  with  RAS 

-  Mission:  Close  Air  Support,  Interdiction  Attack,  Suppression  of  Enemy  Air  Defenses 

-  Enemy:  Vehicles,  Personnel,  Vehicles  with  Countermeasures 

-  Troops:  rotary  wing,  fixed  wing,  unmanned  aeriel  systems 

-  Terrain  and  Weather:  Terrain:  Urban,  Mountain,  Desert,  Jungle.  Weather:  Clear,  Dust, 
Fog,  Rain,  Snow. 

-  Time:  Clock  Hours:  Day,  Night,  Crossover.  Time  Available:  Deliberate,  Hasty  Planning 

-  Civil  Considerations:  Positive  ID,  Weapons  Free,  Collateral  Damage  Considerations. 


•  RAS  Munition  -  Seeker 

-  Mission:  All 

-  Enemy:  Vehicles,  Personnel,  Vehicles  with  Countermeasures 

-  Troops:  All 

-  Terrain  and  Weather:  Terrain:  All.  Weather:  Clear,  Dust,  Fog,  Rain,  Snow. 

-  Time:  Clock  Hours:  Day,  Night,  Crossover.  Time  Available:  Deliberate,  Hasty 
Planning 

-  Civil  Considerations:  Positive  ID,  Weapons  Free,  Collateral  Damage  Considerations. 
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Strategy  Benefits 


MBT&E  enables  Strategy  Development  to 

-  Prioritize  by  End-Results,  Mission-Task 
effectiveness,  SoS-Task  performance,  and 
Operational  Variables  that  are  most  stressing 
to  System  characteristics  and  performance 

-  Employ  these  most  stressing  circumstances  to 
integrate  technical,  Developmental,  Force 
Development,  and  Operational  testing. 

-Articulate  results  in  the  language  of  the 
Warfighter. 
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MBT&E  Procedure 

Step  10 

Develop  the  Evaluation 

Strategy 
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Evaluation  Strategy  Summary 

Outline  Summary  or  Lens  Chart. 
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Army  Test  and  Evai-lpa'I'ioim  Command 
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MBT&E  Procedure 

Step  1 1 

Develop  the  Evaluation 

Measures 
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Operational  Measures 

—  Air  to  Surface  Attack  (ART  3.3.1  Employ  Lethal  Fire  Support)  {Time  to 
target  observation  <15  minutes.} 

-  Operational  Measure:  %  missions  enemy  is  observed. 

-  OM:  %  missions  course  of  action  is  completed  (Engage  RAS  or  JAAT).* 

-  OM:  Time  to  first  target  observation.* 

-  OM:  Stowed  kills. 

--  Support  SBCT  in  AO  (ART  1.4.1  Conduct  Lethal  Direct  Fire) 

—  Employ  RAS  (RAS  7)  {Positive  control  range  >50  nm.} 

—  Engage  with  RAS  (TO  251-1522  Perform  Firing  Techniques) 

{Pssk  >80%.} 

-  OM:  %  missions  target  is  destroyed.* 

-  OM:  Time  of  engagement  (attack  order  to  target  hit).* 

-  OM:  Operator  rating  of  engagement  procedures. 

-  OM:  Pssk  (demonstrated  and  predicted). 


*  Measure  referenced  in  FM  7-15,  The  Army  Universal  Task  List. 
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Technical  Measures 

•  Aircraft  TDL  [Control  RAS  during  flight.]  {Positive  communication  link 
range  >  60nm} 

•  Technical  Measure:  Average  maximum  positive  control  range. 

•  TM:  RAS  position,  speed  and  attitude  information  accuracy. 

•  RAS  Munition  [1.  Provide  situational  information,  2.  Destroy  target.]  {Prel; 
%  non-essental  function  failure  >  93%.}  {Loiter  time  >  45  minutes.} 

•TM:  In-flight  reliability  (%  non  essential  function  failures)* 

•TM:  Maximum  loiter  time  (predicted  and  demonstrated)* 

•  Seeker  [Provide  situational  images,  acquire  and  track  target.] 
{Minimum  Delta-Temperature.}  {Operate  with  all  semi-active  laser  code 
frequencies.} 

•  TM:  Minimum  delta-temperature.* 

•  TM:  Verification  of  SAL  code  performance.* 

•  TM:  SAL  acquisition  range.* 


*  Measure  also  included  in  materiel  system  performance  specification. 
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MBT&E  Procedure 

Step  12 

Assign  Measures  to  Data 

Sources 
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Link  to  Data  Sources 


Chronologically  linked  to  data  source 
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Link  to  Data  Sources 


Critical  technologies  linked  to  capabilities. 
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Capabilities  assessed  through  continuum  of  T&E 
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MBT&E  Procedure 

Step  13 

Constrained  Operational 

Conditions 
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Constrained  Op  Conditions 

•  RAS  Munition  -  Seeker,  Minimum  delta-T 

-Terrain  and  Weather:  Terrain:  All.  Weather:  Clear,  Dust,  Fog,  Rain, 
Snow. 

-  Identified  Data  Sources:  Tower  Tests,  Captive  Flight,  DT  Flight 
Tests,  HWIL. 

-Fog,  rain  and  snow  not  available  to  be  controlled  during  any  of  these 
tests. 

-T&E  Limitation 

•  Limitation:  Not  able  to  test  the  performance  of  the  seeker  under  fog,  rain 
and  snow. 

•  Impact:  Not  able  to  evaluate  seeker  performance  under  these  conditions. 

•  Current  Mitigation:  Integrated  flight  simulation  (IFS)  may  be  used  to 
analyze  seeker  performance  under  these  conditions. 
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T&E  Alternate  COA 


•  T&E  Limitation:  Not  able  to  test  the 
performance  of  the  seeker  under  fog,  rain 
and  snow. 

•  Recommended  COA 

-  Use  IFS  to  characterize  predicted  seeker 
performance  under  fog,  rain  and  snow 
conditions. 

-  Conduct  chamber  “tower”  test  to  confirm 
degraded  performance  in  worst  condition. 
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MBT&E  Procedure 

Step  14 

Develop  Data  Source 
Requirements 


57 


Data  Source  Requirements 

•  Use  design  of  experiments  given  constrained  operational  conditions. 

•  Engage  with  RAS 

-  OM:  %  missions  target  is  destroyed 

-  OM:  Time  of  engagement  (attack  order  to  target  hit). 

-  Engage  with  RAS  Operational  Conditions 

-  Mission:  Close  Air  Support,  Interdiction  Attack,  SEAD 

-  Enemy:  Vehicles,  Personnel,  Vehicles  with  Countermeasures 

-  Troops:  Rotary  wing,  fixed  wing,  unmanned  aerial  systems 

-  Terrain  and  Weather:  Terrain:  Urban,  Mountain,  Desert,  Jungle. 
Weather:  Clear,  Dust,  Fog,  Rain,  Snow. 

-  Time:  Clock  Hours:  Day,  Night,  Crossover.  Time  Available: 
Deliberate,  Hasty  Planning 

-  Civil  Considerations:  Positive  ID,  Weapons  Free,  Collateral 
Damage  Considerations. 
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DOE  Recommendation 


•  Desire  to  test  with  3  crews 


Option  1:  random  draw  of  3 
crews. 

•  Pro:  Smallest  number  of  runs  (24). 

•  Con:  Can  not  separate  crew 
performance 

Option  2:  Test  each  run  once  for 
each  crew. 

•  Pro:  Can  separate  crew  performance 

•  Con:  Maximum  number  of  runs  (72). 


Mission  Troops 

Terrain 

Time  Available 

SEAD 

UAS 

Desert 

Planned 

CAS 

F/W 

Urban 

Planned 

SEAD 

UAS 

Urban 

Planned 

CAS 

F/W 

Desert 

Hasty 

IA 

R/W 

Urban 

Planned 

IA 

F/W 

Urban 

Planned 

IA 

F/W 

Desert 

Hasty 

SEAD 

R/W 

Desert 

Hasty 

IA 

R/W 

Urban 

Hasty 

SEAD 

F/W 

Desert 

Planned 

CAS 

UAS 

Desert 

Hasty 

CAS 

F/W 

Desert 

Planned 

SEAD 

F/W 

Urban 

Hasty 

IA 

UAS 

Desert 

Planned 

IA 

UAS 

Urban 

Hasty 

CAS 

R/W 

Desert 

Planned 

SEAD 

R/W 

Urban 

Planned 

IA 

R/W 

Desert 

Hasty 

SEAD 

UAS 

Desert 

Hasty 

SEAD 

UAS 

Urban 

Hasty 

CAS 

F/W 

Urban 

Hasty 

CAS 

R/W 

Urban 

Hasty 

IA 

R/W 

Desert 

Planned 

CAS 

UAS 

Urban 

Planned 
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DOE  Recommendation 


Option  3  (recommended):  Partial  factorial  using  crews. 

•  Pro:  Medium  set  of  runs  (48). 

•  Pro:  Can  separate  crew  performance  to  acceptable  level. 

•  Con:  More  than  minimum  set  of  runs. 


Mission  Troops 

Terrain 

Time  Available 

Crew 

Mission 

T  roops 

Terrain 

Time  Available 

Crew 

1 

SEAD 

F/W 

Desert 

Planned 

C 

25 

CAS 

F/W 

Desert 

Planned 

A 

2 

CAS 

R/W 

Desert 

Hasty 

B 

26 

IA 

R/W 

Desert 

Hasty 

B 

3 

SEAD 

R/W 

Desert 

Planned 

B 

27 

CAS 

R/W 

Urban 

Planned 

A 

4 

IA 

F/W 

Desert 

Hasty 

C 

28 

IA 

UAS 

Desert 

Planned 

C 

5 

SEAD 

UAS 

Urban 

Hasty 

A 

29 

SEAD 

F/W 

Urban 

Hasty 

C 

6 

IA 

F/W 

Desert 

Planned 

A 

30 

CAS 

UAS 

Desert 

Planned 

B 

7 

SEAD 

UAS 

Desert 

Hasty 

B 

31 

IA 

UAS 

Urban 

Hasty 

C 

8 

CAS 

F/W 

Desert 

Hasty 

C 

32 

SEAD 

F/W 

Desert 

Hasty 

B 

9 

IA 

R/W 

Urban 

Planned 

B 

33 

CAS 

UAS 

Urban 

Planned 

C 

10 

CAS 

R/W 

Urban 

Planned 

B 

34 

CAS 

R/W 

Desert 

Hasty 

A 

11 

SEAD 

R/W 

Urban 

Planned 

C 

35 

CAS 

R/W 

Desert 

Planned 

C 

12 

IA 

UAS 

Desert 

Hasty 

A 

36 

IA 

R/W 

Urban 

Planned 

C 

13 

IA 

UAS 

Urban 

Hasty 

B 

37 

SEAD 

F/W 

Urban 

Planned 

A 

14 

CAS 

F/W 

Urban 

Planned 

C 

38 

CAS 

UAS 

Desert 

Hasty 

C 

15 

SEAD 

UAS 

Urban 

Planned 

B 

39 

SEAD 

UAS 

Desert 

Planned 

A 

16 

SEAD 

F/W 

Desert 

Hasty 

A 

40 

IA 

UAS 

Urban 

Planned 

A 

17 

CAS 

F/W 

Urban 

Hasty 

A 

41 

IA 

F/W 

Urban 

Planned 

C 

18 

SEAD 

R/W 

Urban 

Hasty 

B 

42 

CAS 

UAS 

Desert 

Planned 

A 

19 

SEAD 

UAS 

Urban 

Planned 

C 

43 

SEAD 

R/W 

Desert 

Hasty 

C 

20 

IA 

UAS 

Desert 

Planned 

B 

44 

CAS 

F/W 

Desert 

Planned 

B 

21 

IA 

R/W 

Desert 

Planned 

A 

45 

CAS 

UAS 

Urban 

Hasty 

B 

22 

CAS 

R/W 

Urban 

Hasty 

C 

46 

IA 

F/W 

Urban 

Hasty 

B 

23 

SEAD 

F/W 

Urban 

Planned 

B 

47 

IA 

R/W 

Urban 

Hasty 

A 

24 

CAS 

UAS 

Urban 

Hasty 

A 

48 

SEAD 

R/W 

Urban 

Planned 

A 

MBT&E  Procedure 

Step  15 

Develop  T&E  Databases 
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MBT&E  Toolbox 
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MBT&E  Procedure 
Steps  16-19 

Determine  the  Results  & 
Report  the  Results 
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Report  Example 
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Case  Study  Results 
(Fictional) 

Summary  Results 
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Close  Air  Support 

Time  to  first  target  observation 

<  15  min 
(13  min) 

14.6  min 

Stowed  Kills 

NC 

Predicted:  3,  6,  12 
Demonstrated:  2,  5,  11 

Employ  RAS 

Rating  of  control 

NC 

4.7/5  Excellent 

A/C  TDL 

Positive  Control  Range 

50  km 

62  km 

RAS  Munition 

Avg  Max  Loiter  Time 

45  min 

52  min 

Engage  with  RAS 

%  missions  target  is  destroyed 

NC 

84% 

Time  of  Engagement 

<  15  min 
(2  min) 

1.6  min  i 

Pssk 

80% 

P:  76%,  D:  69% 

RAS  Munition 

In-flight  Reliability 

93% 

82% 

Guidence  and  Control 

Ph/s  Predicted 

90% 

95% 

Ph/s  Observed 

NC 

84% 

Warhead 

Pk/h  Predicted 

95% 

97% 

Pk/h  Observed 

NC 

100% 

Employ  Lethal  Fire  Support 

•  Able  to  arrive  in  the  engagement  area  within  15 
minutes. 

•  Able  to  arrive  in  the  engagement  area  and  destroy 
the  target  within  16.2  minutes. 

•  Able  to  destroy  from  3  to  1 2  targets  per  sortie  (2 
aircraft)  based  on  load  out. 
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Employ  RAS 

•  Able  to  employ  the  RAS  up  to  a  range  of  62 
km  with  a  loiter  time  of  52  minutes. 


Engage  with  RAS 

•  Able  to  engage  the  target  within  2  minutes. 

•  Able  to  engage  and  destroy  targets  with  a  probability 
of  single  shot  kill  of  76%,  +/-  4%. 


NC: 


No  defined  criteria. 
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MBT&E  Procedure 

Steps  19 

Report  the  Results 
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Evaluation  Report 

•  Effectiveness 

Employ  Lethal  Fire  Support 

•  Able  to  arrive  in  the  engagement  area  within  15  minutes. 

•  Able  to  arrive  and  destroy  the  target  within  16.2  minutes. 

•  Able  to  destroy  from  3  to  12  targets  per  sortie  (2  aircraft)  based  on  load  out. 

•  Able  to  employ  the  RAS  up  to  a  range  of  62  km  with  a  loiter  time  of  52  minutes. 

•  Able  to  engage  and  destroy  targets  with  a  probability  of  single  shot  kill  of  76%,  +/-  4%. 

•  Suitability 

Replace  IR  Coolant  Bottle  (enabling  task) 

Able  to  replace  a  spent  IR  coolant  bottle  within  15  minutes  on  the  flight  line. 

Training  (enabling  task) 

Able  to  task  qualify  and  conduct  RAS  missions. 

Reliability  (enabling  attribute) 

The  RAS  demonstrated  a  reliability  of  82%  (time  to  essential  function  failure).  This 
supported  a  stowed  kill  rate  of  3  to  12  targets  per  sortie  (2  aircraft)  based  on  load  out. 
Maintainability  (enabling  tasks) 

OPTEMPO  was  supported  with  a  mean  time  to  repair  of  1.2  hours  and  anticipated 
stockpiles  at  the  ammunition  supply  point. 
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Evaluation  Report 


•  Survivability 


Electromagnetic  Survivability  (enabling  attribute) 

•The  RAS  was  compatible  with  existing  and  induced  electromagnetic 
environments. 

Information  Assurance  (enabling  attribute) 

•  The  RAS  was  able  to  deny  all  information  operations  attacks. 

Jettison  Launcher  (conditional  task) 

•Jettison  of  the  launcher  was  demonstrated  for  each  load  configuration. 
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Case  Study 


•  Function  Need  Analysis 

-  Gap  1:  Time  sensitive  targets  need  to  be  engaged  within  15  minutes. 

•  High  priority  targets,  once  identified,  need  to  be  destroyed  before  they  have  a  chance 
to  escape  or  hide  in  dense  urban  terrain,  approx  15-20  minutes. 

•  Current  aviation  unit  response  time  from  call  for  fires  to  prosecuted  target  as  much  as 
45  minutes,  depending  on  current  location  and  location  of  the  support  call  (50  nm). 

•  Current  quick-response  call  for  fires  from  other  assets  (artillery  and  current  air- 
launched  munition  set)  not  reliably  available  or  not  desired  due  to  need  to  minimize 
collateral  damage  in  dense  urban  terrain. 


Employ  Lethal  Fire  Support 

•  Able  to  arrive  in  the  engagement  area  within  15  minutes. 

•  Able  to  arrive  and  destroy  the  target  within  16.2  minutes. 

•  Able  to  destroy  from  3  to  12  targets  per  sortie  (2  aircraft)  based  on  load  out. 

•  Able  to  employ  the  RAS  up  to  a  range  of  62  km  with  a  loiter  time  of  52  minutes. 

•  Able  to  engage  and  destroy  targets  with  a  probability  of  single  shot  kill  of  76%,  +/-  4%. 
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Case  Study 


•  Function  Need  Analysis 

-  Gap  2:  Immediate  response  (<15  minutes)  and  extended  surveillance  (>45 

minutes)  needed  to  develop  situational  intelligence. 

•  Success  seen  in  using  pre-planned  reconnaissance/surveillance  assets  (RSAs)  to 
observe  suspicious  behavior  of  initial  target  leading  to  other,  more  valuable,  targets. 

•  Currently,  initial  targets  are  being  lost  due  to  response  time  from  observation  to  re¬ 
tasking  of  RSAs  and  inability  of  ground  units  to  continue  to  surveil  initial  targets  in 
dense  urban  terrain  unobserved.  Most  targets  lost  within  15  minutes. 

•  Currently,  targets  taking  as  much  as  45  minutes  to  move  from  initial  observation  area 
to  centralized  base.  (Based  on  pre-planned  RSA  missions.) 


Employ  Lethal  Fire  Support 

•  Able  to  arrive  in  the  engagement  area  within  15  minutes. 

•  Able  to  arrive  and  destroy  the  target  within  16.2  minutes. 

•  Able  to  destroy  from  3  to  12  targets  per  sortie  (2  aircraft)  based  on  load  out. 

•  Able  to  employ  the  RAS  up  to  a  range  of  62  km  with  a  loiter  time  of  52  minutes. 

•  Able  to  engage  and  destroy  targets  with  a  probability  of  single  shot  kill  of  76%,  +/-  4%. 
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MBT&E  Tutorial 

Discussions 


Questions 

Answers 
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MBT&E  Point  of  Contact 


Christopher  Wilcox 

US  Army  Evaluation  Center 
ATTN:  TEAE-AVA  (Mr.  Chris  Wilcox) 
4120  Susquehanna  Ave. 
Aberdeen  Proving  Ground,  MD  21005 

Office:  (410)  306-0475 
Mobile:  (410)  652-0168 
Fax:  (410)306-0398 

chris.wilcoxl  @us.army.mil 
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UNITED  STATES  ARMY  EVALUATION  CENTER 


Mission-Based  T&L 

Tutorial,  2  March  2009 


25th  Annual  NDIA 
T&E  Conference 


Chris  Wilcox 

US  Army  Evaluation  Center 
410-306-0475 

chris.wilcox1@us.army.mil 
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Purpose 


•  To  present  and  overview  the  MBT&E 
methodology  (framework  and  process). 

•  To  engage  in  question/answer  discussions 
on  the  MBT&E  methodology. 

•  To  obtain  audience  feedback  on  the 
MBT&E  methodology. 
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1 


Agenda 

1330:  Mission-Based  T&E  Background 
1340:  MBT&E  Framework 
1400:  Case  Study  Introduction 
1415:  Steps  1-5:  Understand  the  Mission 

1500-1515:  Afternoon  Break 

1515:  Steps  6-8:  Understand  the  System 

1545:  Steps  9-15:  Design  the  T&E 

1615:  Steps  16-19:  Determine  &  Report  the  Results 

1 630:  Discussions/Questions/Answers 


3 


Background  -  Why  MBT&E? 

•  Because  we  were  asked  to... 

-  DA/OSD-level  guidance: 

•  Address  recent  policy  initiatives,  such  as:  Section  231  Report; 
DOT&E/OUSD(AT&L)  T&E  Policy  Revisions  memo;  etc. 

-  “Show  impact  of  materiel  system  strengths/weaknesses  on  the 
operational  capabilities.” 

-  “Integrate  DT  and  OT  and  make  use  of  all  available  data.” 

•  Address  goals,  strategies  and  initiatives  in  DUSA-TEO  Strategic 
Plan,  2007. 

-  “Continuously  improve  T&E  policy  and  procedures.” 

-  “Increase  operational  realism  in  developmental  tests  to  improve  the 
likelihood  of  successful  operational  tests.” 

•  New  TEMP  format  and  DoD  5000  changes. 

-  “Integrated  T&E”  chapter  vs.  DT  and  OT  chapters. 
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Background  -  Why  MBT&E? 

•  Because  we  want  to  improve: 

-  The  way  we  do  our  job. . . 

•  Enable  robust  T&E  strategy  development  for  Joint 
networked  system-of-systems. 

-  The  way  we  support  the  warfighter. . . 

•  Answer  the  “so  what”  question.  (Complete  feedback 
loop  to  Capability-Based  Analysis.) 

•  Develop  way  to  link  system  performance  to  unit  and 
higher  unit  task  capabilities. 


-  The  way  we  support  the  materiel  developer. . . 

•  Scope  T&E  effort  earlier  in  the  acquisition  cycle. 
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MBT&E  Implementation 


Preliminary  Design  Feb  08:  1st  MBT&E  Summit  <C  [egin  Pilot  Projects  (3) 


80%  Design 


May  08:  Community  review  <C  d  Pilot  Projects  (10) 


Final  Design 


Aug  08:  2nd  MBT&E  Summit 


Ongoing  Pilots  (15) 


Published  Procedure^  Dec  08:  Procedure  Review 


Jan  09: 


New  T&E  Concepts  Using  MBT&E 


Lessons  Learned: 

MBT&E  framework  providing  context  of  operational  capability. 
MBT&E  process  is  executable  with  current  personnel  skill  set. 
Efficiencies  can  be  increased  through: 

•  Improved  tools  (templates,  IT,  training,  etc.);  and 

•  Combat  and  materiel  developer  participation. 


Background  -  MBT&E  Overview 

Mission-Based  Test  and  Evaluation 

is  a  methodology  that  focuses  T&E  on  the 

capabilities  provided  to  the  warfighter.  It  provides  a 
framework,  procedure  and  complexity  constraint 

strategies  to: 

-  link  capabilities  to  the  attributes  of  the  materiel  system- 
of-systems; 

-develop  evaluation  measures  that  assess  capabilities  and 
attributes; 

-and  link  the  evaluation  measures  to  all  available  data 
sources. 
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Agenda 

1330:  Introduction  and  Mission-Based  T&E  Background 

1340:  MBT&E  Framework 
1400:  Case  Study  Introduction 
1415:  Steps  1-5:  Understand  the  Mission 

1500-1515:  Afternoon  Break 

1515:  Steps  6-8:  Understand  the  System 
1545:  Steps  9-15:  Design  the  T&E 
1615:  Steps  16-19:  Determine  and  Report  the  Results 
1 630:  Discussions/Questions/Answers 
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Framework  Building  Block 

Capability1  -  The  ability  to  achieve  a  desired  effect 
[or  result,  outcome,  or  consequence  of  a  task2]  ... 


-  under  specified  standards  and  conditions 

-  through  a  combination  of  means  and  ways 

-  to  perform  a  set  of  tasks. 


1. CJCSI  31 70.01  F,  May  2007 

2.  Taken  from  JP  1-02,  Mar  2007,  definition  of  effect.  9 


Framework  -  Task  Hierarchy 


MBT&E  Framework  —  v2 


Process/Products 


Commander’s  Task 
to  Subordinates 


- 1 - 

Transition  to  Allocating  Mission  Means 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


Commander’s  Task 
to  Subordinates 


- 1 - 

Transition  to  Allocating  SoS  Means 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


System  Attributes 


Transition  to  Allocating  Materiel  Means 

± 


Systems  Engineering 

•  Functional  Baseline 

•  Allocated  Baseline 

•  Product  Baseline 


f  AMSAA 

Excellence  in  Analysis 


Capability  = 


Set  of  Tasks 

+ 

Desired  Result 

Desired 
End  State 


Enables 


Operations  (Mission  Tasks) 

^ 

•  UJTLs 

Desired  Mission 

•  Service  TLs 

Task  Results 

•  Implied  Tasks 

Mission  Task  Capability 


System-of-Systems  Tasks 
Service  TLs 
Implied  Tasks 
Collective/Individual  Tasks 


Desired  SoS 
Task  Results 


T 


SoS  Task  Capability 


System  Performance 
•Functions  (shall  do) 
•  “shall  be’s” 

System  Performance 


Desired  System 
Performance  Results 


>- 


High  Level 
Tasks/Results 


Tasks/Results 
Specific  to 
System 


System 

Functions 

3 
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Case  Study  Introduction 


6-10 
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Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  collect  information. 


understand  •  4  steps  to  understand  the  military  operations,  tasks,  task 
the  mission  capabilities  and  mission  context. _ 


0 


understand  •  2  steps  to  understand  the  components  and  attributes  of 
the  system  the  materiel  system-of-systems. _ 


•  1  additional  step  to  understand  the  mission  and  system 
linkages. 


o 


o 


design  the  •  7  steps  to  design  the  T&E  given  the  mission  and  system 
t&e _ understanding. _ 


determine  •  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
THE  RESULTS  data. 


REPORT 
THE  RESULTS 


1  step  to  format  and  report  the  results. 


ii-i 
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Procedure  - 

Collect  Information  (Step  1) 

Purpose:  Collect  available  information  to  gain  understanding  of: 

-  initiating  capability  gaps, 

-  mission  context  and  operational  conditions, 

-  mission  tasks  and  capabilities, 

-  system-of-systems  design  and  materiel  system  required  attributes. 

What  do  I  do? 

•  Work  through  capabilities  developer  and  materiel  developer  to 
obtain  the  available  information. 

-  Functional  Area,  Needs  &  Solution  Analyses,  Analysis  of  Alternatives, 
Requirements  Documents  (DoDAF  Views),  Threat  Assessments, 
Acquisition  Strategies,  Performance  Specifications,  etc. 

When  am  I  done? 

-  Checklist  of  documents  available/not  available  is  completed. 

-  Available  documents  Archived. 

-  Actions  to  obtain  copies  of  documents  available,  but  not  archived,  are 

presented  to  appropriate  IPTs.  12- 
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Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  understand  the  program  context. 


understand  the  •  4  steps  to  understand  the  military  operations,  tasks,  task 
mission  capabilities  and  mission  context. 


understand  the  •  2  steps  to  understand  the  components  and  attributes  of 
system _ the  materiel  system-of-systems. _ 


•  1  additional  step  to  understand  the  mission  and  system 
linkages. 


design  the  test  •  7  steps  to  design  the  T&E  given  the  mission  and  system 
AND  evaluation  understanding. _ 


determine  the  •  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
RESULTS  data. 


REPORT  THE 
RESULTS 


1  step  to  format  and  report  the  results. 


i 
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Define  the  Mission  Context  (Step  2) 

Relation  to  Framework 

•  High-level  operations/mission/tasks. 

•  Task  desired  end  states/results. 


15 
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Define  the  Mission  Context 

(Step  2) 

Purpose:  Define  the  overall  mission  area  context  that  the 

proposed  materiel  solution  is  being  developed  to  support. 

What  do  I  do? 

•  Determine  Operations/Mission/Tasks 

-  Develop  a  description  of  high-level  operations/mission/tasks  and 
their  desired  end  states/results, 

-  Determine  Joint,  network  and  SoS  construct,  and 

-  Determine  organizational  and  support  unit  construct. 

•  Determine  Operational  Conditions 

-  Determine  the  essential  elements  of  mission,  enemy,  terrain  and 
weather,  troops  and  support  available,  time  available,  and  civil 
considerations  (METT-TC). 

When  am  I  done? 

-  High-level  operations/missions/tasks  with  their  desired  end 
states/results  are  documented. 

-  Operational  conditions  (METT-TC  factors)  are  documented. 


16-22 
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Develop  the  Mission  Tasks  (Step  3) 

Relation  to  Framework 

tasks  and  task  threads. 

Task  desired  end  states/results. 


MBT&E  Framework  -  v2 


Process/Products 


kMSAA 

Excellence  in  Analysis 


Capability  =  Set  of  Tasks  +  Desired  Result 


Commander’s  Task 
to  Subordinates 


- 1 - 

•  Transition  to  Allocating  Mission  Means  ■ 

I 

Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


Operations  (Mission  Tasks) 

^ 

•  UJTLs 

Desired  Mission 

•  Service  TLs 

Task  Results 

•  Implied  Tasks 

23 
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Develop  the  Mission  Tasks  (Step  3) 
Mission  Analysis  References 

•  Reference  authoritative  task  lists 

•  Looking  for  most  applicable  task  reference 

•  Task  Description 

•  Conditions 

•  Standards 

•  References  used  in  case  study 

•  Universal  Joint  Task  List  (UJTL),  CJCSM  3500. 

•  Army  Universal  Task  List  (AUTL),  FM  7-15. 

•  Attack  Helicopter  Battalion  Mission  Training  Plan, 
ARTEP  1-1 12-MTP. 

•  Aircrew  Training  Manual,  TC  1-251. 
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Develop  the  Mission  Tasks  (Step  3) 

Purpose:  Develop  the  required  SoS  mission  tasks  and  link 
these  tasks  to  authoritative  tasks  lists. 

What  do  I  do? 

•  Document/Conduct  Mission  Analysis 

-  Develop  SoS  mission  task  threads  and  alternate  task  threads 
where  applicable.  (MS  project  or  similar  tool  can  be  used.) 

-  Determine  task  desired  end  states/results 

•  Link  to  Authoritative  Task  Lists 

-  Develop  linkages  between  the  tasks  identified  above  and  the 
appropriate  authoritative  task  lists.  (UJTL,  AUTL,  unit  Mission 
Training  Plans,  etc.) 

When  am  I  done? 

-  SoS  mission  tasks  with  their  desired  end  states/results  are 
documented. 

-  Links  to  associated  authoritative  tasks  are  documented.  25-26 
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Procedure 

Develop  Supporting  Tasks  (Step  4) 


Purpose:  Develop  the  required  supporting  tasks  that  enable  the  execution 
of  the  SoS  mission  tasks. 


What  do  I  do? 

•  Determine  Conditional  Tasks  and  their  desired  effects/results. 

-  Conditional  tasks  are  performed  during  a  normal  mission  but  are  only 

required  due  to  some  influencing  condition. 

-  Examples:  avoid  threat  missile ,  extinguish  engine  fire,  reset  network  node, 
etc. 

•  Determine  Enabling  Tasks  and  their  desired  effects/results. 

-  Mission  enabling  tasks  are  conducted  in  order  to  enable  the  SoS  mission 
tasks  (task  developed  in  step  (3))  to  be  performed. 

-  Examples:  train,  deploy,  maintain,  etc. 

When  am  I  done? 

-  Conditional  tasks,  enabling  tasks  and  their  desired  end  states/results  are 
documented. 

-  Links  to  associated  authoritative  tasks  are  documented. 
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Procedure 

Identify  Task  Capabilities  (Step  5) 


Purpose:  Identify  and  associate  the  capabilities  required  to 
execute  the  SoS,  conditional  and  enabling  tasks. 


What  do  I  do? 

•  Identify  Required  Capabilities 

-  Identify  the  capabilities  required  to  support  each  task  with  a 
reference  to  applicable  requirements  documents.  (CDD,  CPD,  etc.) 

•  Associate  Tasks  with  Capabilities 

-  Link  the  capabilities  determined  above  with  the  mission,  conditional 
and  enabling  tasks  determined  in  steps  (3)  and  (4) 


When  am  I  done? 


-  Links  between  the  (SoS,  conditional  and  enabling  tasks)  and  the 
requirements  are  documented. 


29-30 
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Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  understand  the  program  context. 


UNDERSTAND  THE 
MISSION 

•  4  steps  to  understand  the  military  operations,  tasks,  task 
capabilities  and  mission  context. 

UNDERSTAND  THE 
SYSTEM 

•  2  steps  to  understand  the  components  and  attributes  of 
the  materiel  system-of-systems. 

•  1  additional  step  to  understand  the  mission  and  system 
linkages. 

DESIGN  THE  TEST 
AND  EVALUATION 

•  7  steps  to  design  the  T&E  given  the  mission  and  system 
understanding. 

DETERMINE  THE 
RESULTS 

•  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
data. 

REPORT  THE 
RESULTS 

•  1  step  to  format  and  report  the  results.  32 
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Determine  SoS  Components  (Step  6) 

Relation  to  Framework 


*  SoS  Components 

*  SoS  Component  Functions  &  “Shall  be’s” 


Materiel  System  Functions:  An  activity  or  action  the  materiel  system  performs  in  support 
of  a  capability  or  part  of  a  capability. 

Materiel  System  “shall  be”:  An  attribute  the  materiel  system  possesses  that  enables  it  to 
perform  a  function,  for  example,  reliability. 


33 
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Determine  SoS  Components  (Step  6) 


Purpose:  Identify  the  physical  components  of  the  materiel 
system  that  support  the  mission  tasks. 

What  do  I  do? 

•  Develop  a  materiel  system  description  starting  from  the  SoS 
level  and  breaking  down  into  components. 

•  Components  at  the  lowest  level  should  be  able  to  be  linked  to 
identifiable  functions  (shall  do’s)  and  enabling  attributes  (shall  be’s). 

•  Identify  the  functions  and  “shall  be’s”  of  the  materiel 
components. 

•  Option:  Develop  technology  risk  areas  for  pre-MS  B 
systems. 

When  am  I  done? 

-SoS  components  with  their  functions/shall  be’s  are  documented.  3< 
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Procedure 

Develop  System  Attributes  (Step  7) 


Purpose:  Identify  the  materiel  system  attributes  and  associate  them  with  the 
system  components. 


What  do  I  do? 

•  Identify  Attributes  Required 

-  Identify  the  materiel  system’s  attributes  required  to  support  the  component 
functions/shall  be’s  with  reference  to  applicable  requirements  documents. 
(CDD,  CPD,  Performance  Specification,  etc.) 


•  Associate  Components  with  Attributes 

-  Link  the  attributes  determined  above  with  the  system  components 
developed  in  step  6. 


When  am  I  done? 

•  Links  between  the  SoS  components  and  their  required  attributes  are 
documented. 
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37-38 


Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  understand  the  program  context. 


understand  the  •  4  steps  to  understand  the  military  operations,  tasks,  task 
mission _ capabilities  and  mission  context. _ 


understand  the  •  2  steps  to  understand  the  components  and  attributes  of 
system  the  materiel  system-of-systems. 


1 

•  1  additional  step  to  understand  the  mission  and  system 
y/  linkages. 

design  THE  test  •  7  steps  to  design  the  T&E  given  the  mission  and  system 
AND  evaluation  understanding. 

DETERMINE  THE 
RESULTS 

•  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
data. 

REPORT  THE 
RESULTS 

•  1  step  to  format  and  report  the  results.  39 
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Capabilities  to  Attributes  - 

Relation  to  Framework 


*  Link  SoS  mission  task  capabilities  to  SoS  component 
attributes. 


39 
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Procedure 

Associate  Capabilities  with  Attributes 

(Step  8) 

Purpose:  Develop  the  linkages  between  the  task  capabilities 
identified  in  step  (5)  and  the  materiel  system  component 
attributes  identified  in  step  (7). 

What  do  I  do? 

•  Link  the  system  attributes  (functions/shall  be’s)  to  the  task  capabilities. 

•  Determine  how  the  system  components  support  the  task  capability. 

•  Determine  redundant  system  support  capability. 

•  Determine  Mission  Enabling  Attributes 

•  Mission  Enabling  Attributes  are  system  enabling  attributes  that  are  not 
specific  to  a  particular  task  capability  -  they  address  all  tasks. 

When  am  I  done? 

-  Links  between  the  system  components  and  their  supported  tasks  are 

documented.  40_ 

-  Enabling  attributes  of  the  SoS  materiel  components  are  documented. 
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Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  understand  the  program  context. 


UNDERSTAND  THE 
MISSION 

•  4  steps  to  understand  the  military  operations,  tasks,  task 
capabilities  and  mission  context. 

UNDERSTAND  THE 
SYSTEM 

•  2  steps  to  understand  the  components  and  attributes  of 
the  materiel  system-of-systems. 

•  1  additional  step  to  understand  the  mission  and  system 
linkages. 

DESIGN  THE  TEST 
AND  EVALUATION 

•  7  steps  to  design  the  T&E  given  the  mission  and  system 
understanding. 

DETERMINE  THE 
RESULTS 

•  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
data. 

REPORT  THE 
RESULTS 

•  1  step  to  format  and  report  the  results.  42 
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Operational  Conditions  - 

Linking  with  the  Matrix 


MISSION 

SYSTEM 

(Overall  mission(s),  ex.  "ART  3.3  Employ  Fires  to  Influence  the  Will,  Destroy,  Neutralize,  or  Supporess  Enemy  Forces." 

_ 

SYSTEM 

FUNCTION  or  CHARACTERS 

CPD  SYSTEM  TECHNICAL 

PERFORMANCE 

REQUIREMENTS 

1  MISSION  EXECUTION  TASKS  (Effectiveness) 

ENABLING  TASKS  (Suitability) 

CONDITIONAL  TASKS  (Survivability) 

ENABLING 

[Effectiveness  mission  task(s),  ex.  "ART  3.3.1  Conduct 
Lethal  Fire  Support"] 

[Support  Mission  task(s),  ex.  "ART  6.0 
Combat  Service  Support"] 

[Survivability  miss 

ex.  ART  5.3  C 
Survivability  Op 

lion  task(s), 

onduct 

erationsl 

1  [First  enabling  characteristic,  ex.  "Reliability"]  [Capab 

|  [Second  enabling  characteristic,  ex.  "Availability"]  [Ci 

1  [Additional  enabling  characteristics  examples:  Logistic 

isi 

ill 

lit 

\l\ 

2  £  ■£  A 

i 

■[First  sub-task  under  first 

|g^)5  P^r^rm^ire^isstons^in"^"  1 

^Degraded  Mode  on  the  Ml  09A6"]  | 

■[Second  sub-task  under  first 

Ic002  Conduct  Emergency  Missions"]  1 

([Capabilities  under  sub-task,  ex.  "KSA  | 

£ 

^Movements"]  [Capabilities  under 

■[First  sub-task  under  second 

■effectiveness  sub-task,  ex.  ”06-4-  1 

IGOOO  Perform  a  Survivability  Move  1 

((Paladin  Only)"]  [Capabilities  under  | 

£ 

£ 

£ 

First  suitability  sub-task,  ex.  "ART 

sub-task,  ex.  "06-4-C008  Store  1 

Ammunition  (Howitzer  Section)"] 

[Capabilities  under  sub-task] 

£ 

£ 

■[First  survivability  sub-task,  ex. 

■"ART  5.3.1  Protect  the  Friendly 

■Force  in  an  AO"]  [Capabilities 

(under  sub-task]  _ 

■[First  sub-task  under  first  survivability  1 

■sub-task,  ex.  "ART  5.3.2  Conduct  1 

(Nuclear,  Biological  and  Chemical  1 

■Defense"]  [Capabilities  under  sub-  | 

£s 

'f 

£ 

X 

[First  system  end-item,  ex.  "Self  Propelled  Howitzer"] 

or  characteristic,  ex.  "Reliability"].] 

[System  end-item  technical 
requirement,  ex.  "KPP  7.  Howitzer 
Reliability"] 

This 

task... 

X 

[First  end-item  sub-system,  ex.  "Armor"] 

[First  end-item  sub-system 
function  #1 ,  ex.  "Protect 
Crew  and  Systems''] 

[First  end-item  sub-system 
technical  requirement,  ex. 
"KPP2.  Force  Protection"] 

X 

[First  end-item  sub-system 
function  #2,  ex.  "Protect 
Crew  and  Systems"] 

[Additional  technical 
requirements] 

,  A  [Second  end-item  sub-system,  ex.  "Cannon"] 

[Second  end-item  sub¬ 
system  function.] 

[Additional  technical 
requirements] 

^TshowiJjB^ 

Under  these 

i 

riscic  am 

M  % 

/  [First  sub-system  of  Second 

/  end-item  sub-system,  ex. 

/ 

[First  sub-system  of  second 
end-item  sub-system 
function,  ex.  "Control  Tube 

T raverse/T urret  movement"] 

[Additional  technical 
requirements] 

[Additional  technical 
requirements] 

[Additional  technical 
requirements] 

conditions. 

/  [Second  sub-system  of 

second  end-item  subsystem, 
ex.  "Tube  Temperature 
Sensor"] 

etc. 

[additional  technical 
requirements] 

[additional  technical 
requirements] 

is  supported  by  this 

/ 

Etc. 

etc. 

[additional  technical 
requirements] 

component/function . . . 

[Second  end-item,  ex.  "Resupply  Vehicle"] 

[additional  technical 
requirements] 

[ 

r 

etc. 

[Additional  technical 
requirements] 

[Additional  technical 
requirements] 

etc. 

etc. 

[Additional  technical 
requirements] 

etc. 

[Additional  technical  requirements] 

etc. 

[Additional  technical 
requirements] 

[Additional  technical 
requirements] 

L 

etc. 

etc. 

[Additional  technical 
requirements] 

TTZT 
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Procedure 

Unconstrained  Operational  Conditions 

(Step  9) 

Purpose:  Develop  the  unconstrained  operational 
conditions  that  must  be  addressed  through  test 
and  evaluation. 

What  do  I  do? 

•  Determine  the  operational  factors  and  conditions  that  T&E 
needs  to  address  given: 

•  the  task  capability  required  and 

•  the  system  function/shall  be. 

When  am  I  done? 

-  Operational  conditions  (for  the  intersection  between 
mission  task  and  system  components  are  documented. 
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Process 

Develop  the  Evaluation  Strategy 

(Step  10) 

Purpose:  Develop  a  summary  description  of  the 
evaluation  to  support  an  early  strategy 
coordination  and  review. 

What  do  I  do? 

•  Develop  the  early  strategy  review  brief  from 
the  mission,  task,  and  system  worksheets 
developed  in  steps  (2)  through  (9). 

When  am  I  done? 


-  Early  strategy  review  brief  is  prepared. 
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46-47 


Evaluation  Measures  - 

Relation  to  Framework 

*  Task  Capability  Measures 

*  Materiel  Performance  Measures 


48 
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Process 

Develop  Evaluation  Measures 

(Step  11) 

Purpose:  Develop  the  evaluation  measures. 

What  do  I  do? 

•  Develop  measures  supporting  the  evaluation  of: 

•  task  capabilities  (task  capability  measures),  and 

•  system  attributes  (materiel  performance  measures). 

•  Complete  linkages  from  measure  -to-  system  -to-  task. 

•  Develop  linkages  between  measures  and  COIs/Criteria. 

When  am  I  done? 

-  Task  capability  and  materiel  performance  measures  are 
documented. 

-  Operational  conditions  pulled  from  task.  48-50 
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Data 

Relation 


*  Data  Sources 


Sources  - 

to  Framework 


•  Linked  to  Evaluation  Measures 


51 
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Design  the  T&E 

Assign  Measures  to  Data  Sources 

(Step  12) 


Purpose:  Identify  the  sources  of  data  to  support  the  analysis  of 
the  evaluation  measures. 


What  do  I  do? 

•  Assign  one  or  more  data  sources  to  each  evaluation  measure. 

•  Review  data  source  matrix  to  determine: 

•  T&E  execution  risk  by  assessing  critical  data  sources; 

•  Developmental  risk  by  assessing  when  critical  technologies  are 
demonstrated;  and 

•  Determine  appropriate  use  of  M&S. 

When  am  I  done? 

-  Data  sources  supporting  each  measure  are  documented. 

-  T&E  effort  and  program  execution  risk  issues  are  identified  and 

coordinated  with  the  appropriate  IPTs.  52-53 
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Design  the  T&E 

Constrained  Operational  Conditions 

(Step  13) 

Purpose:  Develop  the  constrained  operational  conditions  by 
looking  at  the  conditions  that  can  be  addressed  by  the 
identified  data  sources. 

What  do  I  do? 

•  Determine  the  operational  conditions  that  can  be 
addressed  by  the  identified  data  sources.  These  are  the 
“constrained”  operational  conditions. 

•  Determine  the  T&E  limitations  by  comparing  the 
constrained  vs.  unconstrained  conditions  (step  9). 

When  am  I  done? 

-  T&E  limitations  caused  by  the  lack  of  ability  to  address 
all  operational  conditions  documented. 

-  Mitigation  effort(s)  to  lesson  impact  of  T&E  limitations 

are  documented.  54.56 
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Design  the  T&E 

Develop  Data  Source  Requirements 

(Step  14) 


Purpose:  Develop  data  elements  from  each  linked  data 
source. 

What  do  I  do? 

•  Develop  detailed  measure  design. 

•  Determine  data  elements  required  from  the  data  source. 

•  Determine  the  operational  conditions  required  for  each  run, 
sortie  or  sample. 

When  am  I  done? 

-  Data  source  requirements  documented  and  coordinated 
with  the  appropriate  executing  test  agent  (contractor, 
government  range,  independent  test  facility,  M&S,  etc.) 
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Design  the  T&E 

Develop  T&E  Databases  (Step  15) 

Purpose:  Develop  database  architecture  to  enable  efficient 
delivery,  formatting  and  analysis  of  delivered  data. 

What  do  I  do? 

•  Develop  an  evaluation  data  model  from  the  task 
description,  enabling  attribute,  measure  description 
worksheets. 

•  The  evaluation  data  model  is  a  representation  of  the  information 
and  data  assets  required  to  evaluate  the  system  expressed  in 
terms  of  entities  and  relationships  between  entities. 

•  Provide  evaluation  data  model  results  to  the  tester. 

•  The  evaluation  data  model  will  ensure  properly  documented  data 
for  communication  between  the  evaluator  and  the  tester.. 

When  am  I  done? 

-  T&E  database  design  is  documented. 


61 
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MBT&E  Current  Toolbox 


MBT&E  Worksheets 


MBT&E  Matrix 


-  Used  to  build  test  and  evaluation  plan. 


Mission 

Summary 


Evaluation 

Measure 

Description 


Task  and 
Enabling 
Attribute 
Descriptions 


— 

-  Maintains  status  and  tracks 
changes  during  execution. 


62 
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Agenda 


1330:  Introduction  and  Mission-Based  T&E  Background 
1340:  MBT&E  Framework 
1400:  Case  Study 

1415:  Steps  1-5:  Understand  the  Mission 

1500-1515:  Afternoon  Break 

1515:  Steps  6-8:  Understand  the  System 
1545:  Steps  9-15:  Design  the  &E 

1615:  Steps  16-19:  Determine  and  Report  the  Results 
1 630:  Discussions/Questions/Answers 


Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  understand  the  program  context. 


UNDERSTAND  THE 
MISSION 

•  4  steps  to  understand  the  military  operations,  tasks,  task 
capabilities  and  mission  context. 

UNDERSTAND  THE 
SYSTEM 

•  2  steps  to  understand  the  components  and  attributes  of 
the  materiel  system-of-systems. 

•  1  additional  step  to  understand  the  mission  and  system 
linkages. 

DESIGN  THE  TEST 
AND  EVALUATION 

•  7  steps  to  design  the  T&E  given  the  mission  and  system 
understanding. 

DETERMINE  THE 
RESULTS 

•  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
data. 

REPORT  THE 
RESULTS 

•  1  step  to  format  and  report  the  results.  63 
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Execution 

(Steps  16  through  18) 

CONDUCT  TEST  AND  GATHER  DATA  (Step  16) 

Purpose:  Execute  the  planned  data  source  activities  and  gather  the  data  for  analysis. 
What  do  I  do? 

•  Execute  test,  run  M&S,  record  data. 

•  Review  data  for  integrity  and  authentication. 

•  Adjust  T&E  program  based  on  impacts  of  changes  in  schedule  and  system  design. 

PERFORM  DATA  ANALYSIS  (Step  17) 

What  do  I  do? 

•  Data  is  analyzed  according  to  the  procedures  identified  in  step  1 1  and  14. 

•  Performance  results  are  compared  to  standards  identified  in  steps  5  (task  capabilities) 
and  7  (system  attributes). 


GENERATE  EVALUATION  RESULTS  (Step  18) 

What  do  I  do? 

•  Determine  materiel  system  attribute  performance. 

•  Determine  SoS  task  capabilities  and  limitations. 

-  Determine  task  capability  C&L  directly  from  task  capability  measure  results. 

-  Determine  task  capability  C&L  based  on  system  attribute  measure  results. 

•  Determine  task  C&L  impact  on  high-level  mission  task  capabilities. 

-  Determine  ability  to  achieve  desired  end  state  directly  from  capability  measures. 

-  Determine  ability  to  achieve  desired  end  state  from  SoS  task  capability  C&Ls. 


63-65 
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Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  understand  the  program  context. 


understand  the  •  4  steps  to  understand  the  military  operations,  tasks,  task 
mission _ capabilities  and  mission  context. _ 


understand  the  •  2  steps  to  understand  the  components  and  attributes  of 
system  the  materiel  system-of-systems. 


•  1  additional  step  to  understand  the  mission  and  system 
linkages. 


design  the  test  •  7  steps  to  design  the  T&E  given  the  mission  and  system 
and  evaluation  understanding. _ 


DETERMINE  THE 
RESULTS 

•  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
data. 

REPORT  THE 
RESULTS 

•  1  step  to  format  and  report  the  results.  6 
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Mission  Element 


Task  Type 


Definitions  ^ 


Air  Assault  | 

Resupply  a  Maneuver  Brigade 

Mission  execution  tasks. 


-  Tasks  that  describe  a  discrete 
action  that  the  unit  (system  and  its 
operators)  must  perform  in  order  to 
accomplish  its  main  mission. 


Conditional  mission  tasks. 

-  Tasks  that  are  performed  during 
the  mission  that  become  required 
due  to  some  influencing  condition. 

Mission  enabling  tasks. 

-  Tasks  that  enable  the  mission 
execution  and  conditional  tasks  to 
be  performed.  They  usually  occur 
before  or  after  the  mission. 


Conditional  Mission  Tasks 

Level  1  Tasks 

Level  2  Tasks 

1  Acknowledge  Missile  Launch*  I 

t 

Avoid  IR  Threat - i— < 

1  Maintain  SA  of  Threat  I 

Control  Aircraft  Fliqht  I 

1  Deploy  Countermeasures  1 

Mission  Enabling  Tasks 


Leve 11  Tasks  Level  2  Tasks 


< 


Enabling  Attributes 

-  System  attributes  that  affect  all 
tasks. 


Normally 

aggregated 

into 

Effectiveness 

and 

Survivability 


Normally 

aggregated 

into 

Suitability 

66 
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Reporting 

Generate  Evaluation  Report  (Step  19) 


Purpose:  To  generate  the  evaluation  report  which  will  document  the 
evaluation  conclusions. 

What  do  I  do? 

•  Generate  system  performance  and  SoS  task  C&L  conclusions. 

•  Generate  summary  of  key  C&Ls. 

•  Generate  effectiveness,  suitability  and  survivability  conclusions. 

When  am  I  done? 

•  System  performance,  (strengths  and  weaknesses),  and  the  impact  they 
had  on  the  task  capabilities  are  documented. 

•  SoS  task  C&Ls  and  high-leve  task  C&Ls  are  documented. 

•  Overall  summary  of  effectiveness,  suitability  and  survivability  is 
documented. 

67-70 
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Agenda 


1330:  Introduction  and  Mission-Based  T&E  Background 
1340:  MBT&E  Framework 
1400:  Case  Study 

1415:  Steps  1-5:  Understand  the  Mission 

1500-1515:  Afternoon  Break 

1515:  Steps  6-8:  Understand  the  System 

1545:  Steps  9-15:  Design  the  &E 

1615:  Steps  16-19:  Determine  and  Report  the  Results 

1 630:  Discussions/Questions/Answers 


52 


MBT&E  Tutorial 

Discussions 


Questions 

Answers 
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System  of  Systems  Survivability, 
Lethality,  Vulnerability  Assessment 

(SoS  SLVA): 

Ballistic  Vulnerability  Modeling 
Demonstration 
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To  present  a  concept  of  a  System  of 
Systems  SLVA  and  a  demonstration  to 
support  methodology  development. 
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•  Concept  of  System  of  Systems  SLVA 

-  Our  concept  of  a  SoS 

-  System-of-System  Survivability  Simulation 
(S4) 

•  Methodology 

-  New  metrics 

-  Decision  making  process  (DMP)  in  S4 

•  Demonstration  overview 

•  Benefits  to  Test  &  Evaluation  community 

•  Summary 
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RDECOM ^  Qur  Con( 


A  design  connecting 
multiple  levels  of  decision 
makers  and  assets  through 
which  decision  makers  at 
every  level  can  adapt  the 
application  of  their  assets  to 
achieve  their  purpose. 
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of  a  SoS 


he  Physical  Systems: 

e.g.,  Future  Brigade  Combat  Team 
(14+1+1). 

The  Leaders 

Capabilities  conceptualized  as 
combat  power,  a  term  that 
encompasses  all  means  available  to  a 
given  unit  at  a  given  time. 

Leaders  at  the  center,  enabled  by 
information,  execute  the  six  traditional 
warfighting  functions. 

The  Context 

While  we  can  discuss  each  of  the 
above  abstractly,  a  domain  context 
grounds  the  assessment. 

4 


RnFcnn/i'p  prom  Component  to  System  Effects 


•  “Doing  the  correct  thing  well”. 

-  Assesses  an  ability  to  reach  the 
chosen  position  of  attack,  or  to  maintain 
formation  and  arrangement  of  forces, 
etc. 

-  Is  more  about  the  physical  situation, 
and  focuses  more  on  the  internals  of  a 
unit. 

•  “Doing  the  correct  thing” 

-  Traces  the  flow  of  information  (e.g., 
an  enemy  spot  report)  through  the 
network  to  its  consumer  (a  leader); 
thence,  to  an  observable  domain  impact 
upon  a  war  fighting  function. 

-  Is  more  about  the  information 
system,  and  looking  outward  from  a 
unit. 

•  SoS  effectiveness  is  a  joint 
result  these  measurements. 
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SOS 


Explore 


'Configure 


Problem 


Concept 

Wl  I  Wl/l 


A  n  oil  ii>i  r> 

Analysis 
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Process 


I.  Identify  customer 
questions. 

II.  Define  the  concept 
that  addresses 
customer  questions. 

III.  Determine  simulation 
requirements  and 
develop  model 
configuration. 

IV.  Generate  metrics 
from  simulation 
results  . 

V.  Apply  analysis 
methods  to  address 
customer  question. 
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System  of  Systems  Survivability 
Simulation  (S4) 


fSL 


•  S4  is  a  small-unit  force-on-force 
Agent  based  simulation  designed 
to  assess  SoS  effectiveness. 

•  As  an  Agent  based  model,  the 
approach  to  decision  making  is 
very  different  than  current  Army 
force-on-force  models. 

-  Emphasis  is  placed  upon  the 
military  decision  making  processes 
(DMPs)  and  the  communications 
network  that  link  these  DMPs  within  a 
SoS. 

-  Each  DMP  represents  human 
decision  makers  on  the  battlefield  that 
is  dynamically  driven  by  the 
information  available  during  simulation 
execution 


SE  I 


SimulMes 


Base  Plan  :  Team  Infantry  contains  enemy  South  of  PL  Gold 
to  protect  advance  of  Team  MCS  to  Farsi  P.  Team  ftlCS  follows 
Team  Infantry  (or  advances  through  Soda  P.)  to  secure  Farsi 
P.  to  protect  advance  of  C  AG  H  to  airport. 


Assembly 

Area 


I'm  Evaluation  Viewer  f^~]fpfxl 


Filename:  E: \\APS\FinalData\NoREAOverallAPSConfigs\REA-NO-COA1  RunSeries.ncx  Data  Table:  REA-N0-C0A1 -AIIRunSeries  Filter:  current 


Number 

PREMISE 

CONCLUSION 

Support 

|  Confide... 

Lift 

73 

HasAPS  ♦  True 

1  stICVCmbtMLS  •  [9th  Decile,max] 

0.1047... 

0.1309... 

1.25 

55 

AtLeastSlewedAPS  •  True 

1  stICVCmbtMLS  .  [9th  Decile,max] 

0.0857... 

0.2142... 

2.0454.. 

72 

HasAPS  •  True 

1  stICVCmbtMLS  .  [8th  Decile,max] 

0.1904... 

0.2380... 

1.1904.. 

69 

AtMostFixedAPS  .True 

2ndlCVCmbtMLS  .  [8th  Decile,max] 

0.1428... 

0.3571... 

1.7045.. 

15 

APSFixedOnly  *  True 

2ndlCVCmbtMLS  *  [8th  Decile, max] 

0.0857... 

0.4285... 

2.0454. 

1 

APSAII  »  True 

1  stICVCmbtMLS  »  [9th  Decile,max] 

0.0476... 

0.2380... 

2.2727. 

0 

APSAII  .True 

1  stICVCmbtMLS  .  [8th  Decile,max] 

0.0761... 

0.3809... 

1.9047. 

54 

AtLeastSlewedAPS  *  True 

1  stICVCmbtMLS  .  [8th  Decile,max] 

0.1238... 

0.3095... 

1.5476. 

68 

AtMostFixedAPS  .True 

1  stMCSCmbtMLS  .  [min,2nd  Decile] 

0.1238... 

0.3095... 

1.5476. 

< 

Actions 

Save  and  Load  Functions 

File  name  to  save  filter  in 
|c  m AGE /FiltersJDefaultF liter  c 


Analyze  results 


Current  Evaluate 


Prevalen...  |  SE-  |  Odds  R...  |  SE+  \  * 

Infinity  NaN  Infinity  Infinity 

6.75  1.6968...  8.3181...  40.776... 


5  0.7884...  6.25  49.540... 

3.21428...  1.6222...  4.4444...  12.176... 
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•  Concept  of  System  of  Systems  SLVA 

-  Our  concept  of  a  SoS 

-  System-of-System  Survivability  Simulation 
(S4) 

•  Methodology 

-  New  metrics 

-  Decision  making  process  (DMP)  in  S4 

•  Demonstration  overview 

•  Benefits  to  Test  &  Evaluation  community 

•  Summary 
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Roots  of  the  approach — 

The  vulnerability/lethality  "taxonomy 


fSL 


-  I,  ■ 


€ 


Component 

status 


Task-success 

status 


Functional 

status 


operations  research, 
missions,  scenarios,  .. 


engineering, 
criticality  analysis,  ... 


Interactions 


physics, 
penetration  models, 
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Platform  Metrics  for  Demonstration: 

icv 


m. 


(12)  Level  2,  (40)  Level  3  Elements  of  Functional  Degradation  (EFD) ,  and  (2)  Level  4  Loss-of-Functional  (LoF)  Utility 

a* 


Crew 

Mobility 

c1  Commander  Incapacitated 

k-'mM  Reduced  Maximum  Speed  20% 

c2  Squad  Leader  Incapacitated 

m1  2  Reduced  Maximum  Speed  40% 

c3  Driver  Incapacitated 

m1  3  Reduced  Maximum  Speed  60% 

Passengers 

m1  4  Reduced  Maximum  Speed  80% 

p.,  Passenger  1  Incapacitated 

m1  5  Reduced  Maximum  Speed  100% 

p2  Passenger  2  Incapacitated 

m2AA  Reduced  Acceleration  20% 

p3  Passenger  3  Incapacitated 

m2 12  Reduced  Acceleration  40% 

p4  Passenger  4  Incapacitated 

m2 1  3  Reduced  Acceleration  60% 

p5  Passenger  5  Incapacitated 

m2 1 4  Reduced  Acceleration  80% 

p6  Passenger  6  Incapacitated 

m2 1  5  Reduced  Acceleration  100% 

p7  Passenger  7  Incapacitated 

m2  2  -!  Reduced  Steering  20% 

p8  Passenger  8  Incapacitated 

m2.2.2  Reduced  Steering  40% 

m2  2  3  Reduced  Steering  60% 

m2  2  4  Reduced  Steering  80% 

Catastrophic  Loss 

m2.2.5  Reduced  Steering  100% 

/(I  Fuel/Ammo 

m2  3 1  Reduced  Braking  20% 

m2  32  Reduced  Braking  40% 

m2  3  3  Reduced  Braking  60% 

*  A. 

m2  34  Reduced  Braking  80% 

..e\ * 

/T72  3  5  Reduced  Braking  100% 

m2  4  Reduced  Visibility  (driver’s  sensor) 

Loss  of  Function 

m3A  Stop  After  60  Minutes 

MLOF  Mobility  Loss  of  Function 

m3  2  Stop  After  30  Minutes 

FLOF  Firepower  Loss  of  Function 

m3  3  Stop  After  10  Minutes 

m3  4  Stop  After  1  Minute 

Firepower 

f,  Lost  Ability  To  Fire  Buttoned  Up  Main  (RWS) 
f3  Degraded  Initial  Rate  of  Fire  of  Main  (RWS) 
f4  Degraded  Subsequent  Rate  of  Fire  of  Main  (RWS) 
f7  Total  Loss  of  Firepower  Main 
f12  Total  Loss  of  Firepower  Secondary 
Communication 
x11  Reduced  Range  (antenna  loss) 
x1  2  Reduced  Range  (power  amp  loss) 
x2  Lost  Line-of-Sight  (LOS)  Data 
x3  Lost  LOS  Voice 
x4  Lost  Non-LOS  Data 
x7  Lost  External  Communications 
x7A  Lost  Encryption  Capability 
x7  2  Lost  Channel/Frequency  Selection  Capability 

Target  Acquisition  (‘sensing’) 

a1  Lost  Daylight  Acquisition 
a2  Lost  Night  Acquisition 

a3  Lost  Range  Finder  Capability^^^^^_^^^^^ 
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System  capabilities  aggregate 
from  subsystems  and  components 


The  x6  fault  tree. . .  cutting  it  degrades  the  system 
with  lost  internal  communications 


Rear  intercom  3:*** 


7 


Rear  intercom 

_ 2 _ 


Internal  voice  communications 


'll 


Internal  voice  communicatiopg 


I 


/ 

# 

# 

f 


Rear  FFCS 


Driver’s  intercom 

I 


Gunner  FFCS 


1 


Commander’s  intercom 

l 


Wire:  gunner  FFCS  to  MCS 


Gunner’s  intercom 

I 


Slip  ring 


All  Power 


5 


\ 

% 

« 

I 

I 


; 


Wire:  slip  ring  to  rear  FFCS 


Wire:  rear  FFCS  to  MCS 

/ 


1 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


3  MAR  2009 


11 


RDECOM 


S4  Decision  Making  Process  (DMP): 

use  of  EFD  data 


•  Awareness  of  EFDs 

-Perception  Manager 
-Report  Manager 

•  Adaptation 

-  Platform 
-Company 

-  Platoon 
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•  Concept  of  System  of  Systems  SLVA 

-  Our  concept  of  a  SoS 

-  System-of-System  Survivability  Simulation 
(S4) 

•  Methodology 

-  New  metrics 

-  Decision  making  process  (DMP)  in  S4 

•  Demonstration  overview 

•  Benefits  to  Test  &  Evaluation  community 

•  Summary 
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I :  I  dentify  Customer  Questions 


•  How  can  Mission-based  analysis  support  cost  effective 
test  planning,  i.e.,  Live-Fire  shot  selection  and 
Developmental/Operational  T esting? 

-  What  EFD  are  critical  to  operational  testing  (034)? 

-  What  performance  parameters  are  important  to  capture  in 
developmental  testing? 

-  What  are  the  platform  vulnerability  issues  to  assess  with 
MUVES-S2? 

•  How  can  the  impact  of  a  test  event  be  shown  in  a 
mission  context? 

-  What  EFDs  impact  mission  success? 

-  By  contrast,  for  which  EFDs  can  the  unit  compensate? 

•  Can  unknown  SLV  issues  be  revealed  (discovered)  via 
Emulation  involving  adaptive  agents? 
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1 1 :  Configure  the  S4  to  Accept  EFD's 


•  034  in  a  controlled  environment: 

•  Assess  task  execution  to  “attack  by  fire” 

O37 

•  Assess  EFD  impact  on  mission 

Demonstration  objective  was  to 
put  ballistic  damage  into  mission  context. 
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•  Concept  of  System  of  Systems  SLVA 

-  Our  concept  of  a  SoS 

-  System-of-System  Survivability  Simulation 
(S4) 

•  Demonstration  overview 

•  Methodology 

-  New  metrics 

-  Decision  making  process  (DMP)  in  S4 

•  Benefits  to  Test  &  Evaluation  community 

•  Summary 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


3  MAR  2009 


16 


Mission-based  T&E 


MBT&E  Framework  -  v2 


Process/Products 


Commander’s  Task 
to  Subordinates 


Transition  to  Allocating  Mission  Means 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


Commander’s  Task 
to  Subordinates 


Transition  to  Allocating  SoS  Means 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


System  Attributes 


Transition  to  Allocating  Materiel  Means 

i 


Systems  Engineering 

•  Functional  Baseline 

•  Allocated  Baseline 

•  Product  Baseline 


Capability  = 


Desired 
End  State 


Desired  Military 
Condition  Results 


Operations  (Mission  Tasks) 

•  UJTLs 

•  Service  TLs 

•  Implied  Tasks 


Desired  Mission 
Task  Results 


System-of-Systems  T  asks 

•  Service  TLs 

•  Implied  Tasks 

•  Collective/Individual  Tasks 

T 


Desired  SoS 
Task  Results 


SoS  Task  Capability 


Evaluated  by 


Mission  End- 
State  Measure 


^\Enables 

System  Performance 
•Functions  (shall  do) 

•  “shall  be’s” 

Desired  System 
Performance  Results 

- 

System  Performance  Capability 

Task 

>|  Capability 
Measure 


System 

Performance 

Measure 


S4 


MUVES-S2 
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RDECOM > 


New  metric: 
expected 


m  1 . 1 
1 


l,laxto  rfnVf  system 


4'1  axlg  drive,  system 

‘  I  ' 


|  ajjy  single  whee i  suspension  system  | 


|  aity'  bin^lic  tire  sueprnsirjci  sj^ltin  | 

[  3nS  or  4lh  whjnei  drive:  syntnn  right  nr  left 


System  representation  (for  each  variant) 

-  Criticality  analysis: 

-  List  of  elements  of  functional  degradation  (EF 

-  System  (critical  categories  and  EFD)  represei 

-  Identified  tasks. 

-  Task  to  requirement  capability  mapping. 

Model  results  analysis 

-  Cell-by-cell 

-  Probability  of  each  EFD  per  threat. 

-  Probability  of  task  failure. 

-  Bar  charts 

-  Probability  of  each  EFD  per  threat. 

-  Probability  of  task  failure. 

-  View  average  tables 

-  Probability  of  each  EFD  per  threat. 

-  Probability  of  task  failure. 

-  Identified  critical  categories/components  and  EFD  driving  vulnerability. 

Damage  assessment  and  post-shot  analysis  report 

-  Identified  critical  categories/components  and  EFD  driving  vulnerability. 

-  Correlate  ballistic  damage  to  mission  essential  task  failure. 


nnFcoMi h  System  capability  representation  mi 

^  (MAR  2008) 

Crew 

Firepower  T 

Protection 

c,  Squad  Leader  Incapacitated 

Cj  Driver  Incapacitated 
c«  Operator  t  Incapacitated 

Cf  Oparatot  2  Incapacitated 
c.  Gunner  Incapacitated 
ct  Loader  Incapacitated 

Passengers 

Pi  Passengers  Incapacitated 

fj  Degraded  Delivery  Accuracy  o(  Mam 
f,  Degraded  antral  Rate  olFire  of  Main 
/ 4  Degraded  Subsequent  Rate  of  Fee  of  Mam 
r,  Degraded  Maiimien  Range  Mam 
t,  Lost  Reload  CapabiMy 

1,  Total  Loss  of  Firepower  Main 

f.  Lost  Ability  to  Fire  Buttoned  Up  Secondary 

1,  Degraded  Delivery  Accuracy  of  Secondary 
fu  Degraded  Initial  Rate  of  Fee  of  Secondary 

V,  Lost  Nuclear  Delaction 
a,  t  Lost  Biological  Detection 
a, ,  Lost  Chemical  Detection 
a, ,  Lost  Climate  Control 
s,  t  Lost  Over  Pressure  CapabiMy 
a,  Lost  Signature  Management  CapabiMy 
ay,  Lost  Theimal  Signature  Management 
o,j  Lost  Acoustic  Signature  Management 
a,  Lost  Active  Protection  Capability 

Catastrophic  Loss 

Degraded  Subsequent  Rate  of  Fire  of  Secondary 
l„  Total  Loss  of  Firepower  Secondary 
f,s  Loss  of  Fire  Aller  t  Minutes 

a, ,  Lost  Threat  Warning  (cueing)  Capability 
a,^  Lost  Threat  Tracking  Capability  (0  AEU) 

0,3  Lost  Launch  CapabiMy 

Mltlja 

I  «IJ  Al  m02  aU  -ni. 4  rf/U  *14  n  Cl  N  fi  m 
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New  metrics  for  analysis 


For  each  [Platform 

-  Cumulative  time  that  the  platform  spent  with  each  EFD. 

For  each  platform  type 

-  A  count  of  the  total  number  of  hits  on  platforms  of  each  type  by  all  munition  types 

-  Correlation  of  critical  category  to  EFD. 

-  The  absolute  mean  time  a  platform  of  a  given  type  spends  in  each  EFD. _ 


Mean  cumulative 
Cumulative  time 
Cumulative  time 
Cumulative  time 
Cumulative  time 
Cumulative  time 
Cumulative  time 
Cumulative  time 
Cumulative  time 
Cumulative  time 


time  in  each  ECD  for  instance:  ICV-A15 
spent  in  ECD  0  (ml  .1  )  =  0.0  or  0.0  % 
spent  in  ECD  1  (ml  .2)  =  0.0  or  0.0  % 
spent  in  ECD  2  (ml  .3)  =  0.0  or  0.0  % 
spent  in  ECD  3  (ml  .4)  =  364.75  or  29.53  % 
spent  in  ECD  4  (ml. 5)  =  0.0  or  0.0  % 
spent  in  ECD^mgJJj^OjO^rOjOJ^ 
spent  in 
spent  in 
spent  in 


#  hits  on  platform  type  ICV  by  munition  type  ExampleLargeKE  is  3 


P(m3.3  |  hit  by  threat  ExampleLargeKE)  =  0.6666666666666666 
P(m3.2  |  hit  by  threat  ExampleLargeKE)  =  0.6666666666666666 
P(m1.1  |  hit  by  threat  ExampleLargeKE)  =  0.6666666666666666 
P(m3.1  |  hit  by  threat  ExampleLargeKE)  =  0.6666666666666666 
P(m3.4  |  hit  by  threat  ExampleLargeKE)  =  0.6666666 
P(m2.1.2  |  hit  by  threat  ExampleLargeKE)  =  0.66666( 

P(m2.1.3  |  hit  by  threat  ExampleLargeKE)  =  0.66666( 

P(m2.1.4  |  hit  by  threat  ExampleLargeKE)  =  0.66666( 

P(m2.1.5  |  hit  by  threat  ExampleLargeKE)  =  0.66666( 

P(m2.2.2  |  hit  by  threat  ExampleLargeKE)  =  0.66666( 


P(m1.3  |  hit  by  threat  ExampleLargeKE)  =  0.6666666 


Results  for  cc  antenna  (1 )  vs  EFD  ml  .2  (1 ): 

Sample  size  =  5 
Prob  (Y  |  X)  =  0.75 
Raw  data 
1  0 
1  3 

mean  and  std  dev  for  X  =  0.8  0.39999999999999997 
mean  and  std  dev  for  Y  =  0.6  0.4898979485566356 
Covariance  of  X  and  Y  =  0.12 
Correlation  of  X  and  Y  =  0.6123724356957946 


Mean  cumulative  time  in  each  EFD  for  all  targets  of  type:  ICV 
Cumulative  time  spent  in  EFD  0  (ml  .1 )  =  388.0  or  38.8  % 
Cumulative  time  spent  in  EFD  1  (ml  .2)  =  388.0  or  38.8  % 
Cumulative  time  spent  in  EFD  2  (ml  .3)  =  388.0  or  38.8  % 
Cumulative  time  spent  in  EFD  3  (ml  .4)  =  388.0  or  38.8  % 
Cumulative  time  spent  in  EFD  4  (ml.5)  =  388.0  or  38.8  % 
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•  Concept  of  System  of  Systems  SLVA 

-  Our  concept  of  a  SoS 

-  System-of-System  Survivability  Simulation 
(S4) 

•  Methodology 

-  New  metrics 

-  Decision  making  process  (DMP)  in  S4 

•  Demonstration  overview 

•  Benefits  to  Test  &  Evaluation  community 

•  Summary 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


3  MAR  2009 


20 


•  Integration  of  higher  fidelity  V/L  data  within  the 
SoSA  process  has  been  demonstrated. 

•  DMPs  have  been  enhanced  to  utilize  additional 
information  provided  by  higher  fidelity  V/L  data. 

•  In  light  of  the  Mission-based  T&E  strategy,  the 
community  can  benefit  from  higher  fidelity  V/L 
data  and  SoSA  capability  development  in  SLAD. 
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Introduction 


•  746th  Test  Squadron  Capabilities 

•  Navigation  Test  and  Evaluation  Laboratory 
(NavTEL)  Capabilities 

•  Conventional  Test  Process 

•  Automated  Test  Process 

-  Components 

-  Flow  &  Layout 

-  Receiver  Support 

•  Automated  Test  Process  Example 

•  Future  Applications 
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746th  Test  Squadron 

Central  Inertial  &  GPS  Test  Facility  (CIGTF) 


•  Premier  Facility  for  System  and  Component  Level  Testing 

-  Missile  Guidance  and  Control  Systems 

-  Inertial  Navigation  Systems  (INS) 

-  Global  Positioning  System  (GPS)  Receivers 

-  Embedded  GP S/INS  (EGI)  Navigation  Systems 

-  Systems  Integration 

-  Pointing  and  Tracking  Systems 

-  Joint  UAV  Testing 

•  Laboratory  Tests  Target  Variables  Prior  to  Field  Testing 

•  Reference  and  Field  Tests  Verify  Lab  Results 


UNCLASSIFIED 


NavTEL 


«...  Flight  Profile 
Platform  Models 


Navigation  Test  &  Evaluation  Laboratory  (NavTEL) 

-  Hardware-in-the-Loop  Design 

-  Trajectories  (Real  &  Simulated  Using  AGI’s  STK) 

-  Models:  Scenario  Dependent  (Sensors  &  Control) 

-  GPS  Simulators  (Spirent  GSS  7700)  Modem  Signals 

-  EGI  Simulator  (Spirent  Simlnertial) 

-  Interference  Signal  Generators  (Jammers) 

-  Wave  Front  Simulators  (Multi-Element  Antenna  Test) 
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Conventional  Methodology 


•  Spirent  GSS7700  Simulator 

-  Setup  individual  scenarios 

-  User  needs  to  be  present  to  start  new  scenarios 

•  DOS  based  PC/ 104  Data  Acquisition  System  (DAS) 

-  Limited  number  of  receivers 

-  Use  of  removable  media  for  data  and  Operating  System 

•  Must  be  off  to  access  data 

•  Must  be  reloaded  whenever  media  is  cleared 

•  Must  be  reconfigured  every  time  a  hardware  change  is  made 

-  User  needs  to  be  present  to  record  new  data  set 


UNCLASSIFIED 


N 


Conventional  Process 


Spirent  GSS7700  GPS  Simulator 


PC/104  Data  Acquisition  System 
(DAS)  for  Receiver  Collection 
and  Control 


GPS 

RF  Signal 


DOS> 

DOS>  cd  Navix 
DOS>  dir 
DOS>  dfkjdfkskfk. 
ksfkfjd,  lfjksdfkd 
DOS> 


DOS  Prompt 


Unit  Under  Test  (UUT): 
GPS/INS  Receivers 


-i 


8  F*j|  gA 


1  ^ 


a 


•dP 


Disk 


Analysis 
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Motivation 


•  Control  large  numbers  of  receivers 

•  Data  collection  from  large  numbers  of  receivers 

•  Data  transfer  without  affecting  data  collection 

•  Parametric  Analysis  -  Characterize  GPS  Receivers 

-  Perform  Design  of  Experiments  (DOE)  on  variables 

-  Automated  scenario  loading 

-  Automated  data  collection  and  analysis 
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Motivation  Cont . . 


•  Simulation  Analysis  -  Recreate  Field  Test  Events 

-  Read  in  motion  files  and  create  MOD  files  to  create  the 
correct  signal’s  characteristics 

-  Create  a  software  model  that  can  be  used  to  describe  a 
receivers  behavior 

•  Solution:  Remove  the  operator 

-  “Human-on-the-loop”  -not  in  it 
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Automated  Components 


•  MOCHA  •  SPIDAR 


•  SimRemote  •  EASI 


UNCLASSIFIED 


MOCHA 


•  Multiple  Operation  Central  Host  Application  (mocha) 

-  Utilizes  multiple  Quatech  Serial  Device  Servers 

-  Seamless  Connectivity 

-  Seamless  Data  Collection 

-  Real-Time  Monitoring 

-  Quick  Initialization 

-  Remote  Scripting  mocha 


Serial  Device  Server 
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SimRemote 


•  SimRemote 


-  Extension  of  SimGEN  software 

-  Remote  interface  for  Spirent  Simulator  control 

-  Provides 


•  Automated  scenario  loading 

•  Truth  data  monitoring 


SimRemote 


UNCLASSIFIED 


SPIDAR 


•  SPIDAR 


-  MATLAB  based  script 

•  Easy  to  modify  for  new  tests 

-  Generates  Spirent  test  scenarios 

-  Allows  parametric  testing  of  multiple  GPS  variables 


Parametric  Tests 


Spirent  Simulation 
Scenarios 
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EASI 


■ 


•  Easy  Array  Software  Interface  (EASI) 


-  Glue  Program  for  MOCHA  and  SimRemote 

-  Digests  Spirent  scenarios  easi 

-  Records  MOCHA  summary  data 

-  Records  simulator  truth  data 


MOCHA 


SimRemote 
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Process  Flow  Chart 


7700 


Parametric  Runs 


•t  +  % 


Scenario  Generation 


Scenario  Name  -  Test  i 


Data  Collection 

_ t 


Pretalen’s  Spirent 
Data  Generator 
(SPIDAR) 


Scenario  | 
4  lists  \ 


Pretalen’s  easy 
array  software 
interface  (EASI) 


SimRemote 

commands 


Receiver 


746  TS’s  receiver 
interface  MOCHA 


[Inputs 


CO 


Initialization 


Receiver  Test  Array  (1-8  receivers) 
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Process  Layout 
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Receiver  Support 


•  Communication  I/O  Standards 

•  Output  Data  (from  the  receiver) 

-  Used  for  real  time  display 

-  Recorded  for  post-processing  and  analysis 

•  Input  Data  (sent  to  the  receiver) 

-  MOCHA  sends  commands  to  receivers  with  this  data 

•  Code  type  to  track 

•  Initialize  position 

•  etc... 
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Supported  Receivers 


•  MOCHA  modules  currently  support: 

-  Military  (ICD-GPS-153  on  RS-232  /  RS-422) 

•  PLGR,  DAGR,  GB-GRAM... 

-  Civilian 

•  Javad  Receivers 

•  NMEA  data  only 
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Future  Receiver  Support 


•  MOCHA  modules  are  being  developed  to  support: 

-  Military  Receivers 

•  MAGR,  MAGRU,  MAGR2K,  R3A(046) 

•  Force  5,  GEM  III,  GEM  IV  . . . 

-  Civilian  Receivers 

•  Garmin  V,  iiblox,  Ashtech  Z. . . 
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Automated  Test  Process  Example 


SPIDAR:  generates  500  scenarios 


EASI  (ControiySimRemote  Software  1 

•Read  File  Listing  Scenarios  i 

•Initialize  Rcvr  /  Send  Run  name  &  #  j 
•Arm  scenario  /  Command  to  Start  1 
•Scenario  ends  -  increment  &  repeat  i 
•Set  thresholds  for  events  j 

•Compare  data  &  write  summary  report' 


and  a  list  of  scenarios 


SimRemote:  GPS  Simulator 

•Start  Simulation 

•Runs  until  running  scenario 

is  complete 

•Wait  for  EASI  to  send  next 
scenario 

•Sends  real-time  truth  data 


MOCHA:  Data  Collection  S/W 

•Collects  raw  receiver  data 
•Read  msgs  3,4,5040 
•Sends  real-time  performance 
data  to  EASI 


to  EASI 


7700 


oo 


Power 


UUT1 


Scenario  Generation/Controller 


Router 


Report 


Data  Collection  &  Data  Analysis 
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Example  Cont, , . 


•  Ran  500  scenarios  each  between  5-15  minutes  long 

•  >  66  hours  of  GPS  simulations 


•  Completed  on  8  DAGRS  in  <  3  days 
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Future  Applications 


•  Advantages  of  Automated  Test  Process 

-  Support  multitude  of  receivers 

-  Intuitive/Simple  GUI  monitoring  &  control 

-  Quick  access  to  multiple  receiver  functions 

•  Applications  of  Automated  Test  Process 

-  GPS  End-to-End  Testing 

-  Satellite  Reference  Station 

-  Dynamic  FlightA^an  Testing 
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Summary 


•  746th  Test  Squadron  Capabilities 

•  Navigation  Test  and  Evaluation  Laboratory 
(NavTEL)  Capabilities 

•  Conventional  Test  Process 

•  Automated  Test  Process 

-  Components 

-  Flow  &  Layout 

-  Receiver  Support 

•  Automated  Test  Process  Example 

•  Future  Applications 
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746th  Test  Squadron 


Questions? 
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Evaluating  the  Mission  -  Translating 
System  Performance  to  Unit  Capabilities 


NDIA  National  Test  & 
Evaluation  Conference 

Atlantic  City,  NJ 


RdundTable 
Defense,  llc 


w  Evaluating  the  Mission 


March  2009 


Agenda 


•  Purpose 

•  Background 

•  MBT&E  Fundamentals 

•  SE  Approach 

•  Example 

•  Summary 


Evaluating  the  Mission 


Purpose 


•  Identify  a  Systems 
Engineering  Approach 
that  may  be  useful  in 
translating  system 
performance  to  unit 
capabilities 

•  Present  a  methodology 
that  integrates  the  test 
&  evaluation  function 
with  requirements 
analysis  and  materiel 
development 


Evaluating  the  Mission 


March  2009 


Background 


•  Some  factors  driving  MBT&E: 

-  McQueary-Young  Memo  (Dec  2007) 

-  Section  231  Report  (July  2007) 

-  CJCSI  31 70.01  F  JCIDS  Process 

•  ATEC,  MCOTEA,  COMOPTEVFOR  and  AFOTEC 
developing  approaches 

-  Similarities  in  the  Mission  Task  identification  and 
decomposition  process 

-  Key  differences  are  in  the  complexity  of  the  evaluation 
methodology 

•  Integrated  Testing  (DT  &  OT)  provides  a  continuum 
of  knowledge  throughout  System  development 


Evaluating  the  Mission 


March  2009 


Integrated  Test  Approach 


MS  B  OTRR 


MBT&E  Fundamentals 


•  In  order  to  gather  data  that  accurately  answer  the  Critical 
Operational  Issues  (COI)  and  illustrate  capabilities  and 
limitations  of  the  system,  the  test  process  must  begin  and  end 
with  a  paradigm  that  ties  system  Attributes  to  operational  tasks 
or  missions  at  the  unit  level. 

•  Mission  Based  Test  &  Evaluation  (MBT&E)  represents  a  thought 
process  to  guide  the  evaluators  in  developing  the  T&E  strategy 

-  Must  take  advantage  of  work  done  before  by  other  agents  in  the 
Acquisition  process 

-  Understanding  the  documented  missions  for  the  System  vice 
recreating  mission  and  task  analysis 

•  Definitions  (For  the  purpose  of  this  brief) 

-  Effectiveness  -  Capability  of  the  Unit  to  accomplish  the  Mission 

-  Suitability  -  Factors  that  Impact  the  Unit’s  Mission  Capability 
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SE  Approach 


What  Mission  Tasks  was  the 
System  developed  to 
perform? 

What  System  Functions  are 
required  to  perform  those 
Tasks? 

What  Attributes,  defined  in 
the  CDD  and  delivered  in  the 
System,  enable  the  function? 

Which  measures  in  the 
Evaluation  Matrix  were 
identified  in  the  JCIDS 
process  that  convinced 
leadership  to  develop  the 
System? 
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JCIDS  Gap  Analysis  Process 


Strategic  Policy 
Guidance 


Capstone  Concept  for  Joint  Operations 


FAA 

Cm  nptiAno  1 

Joint  Operating  Concepts  (JOC) 

Joint  Functional  Concepts  (JFC) 

UIILllOlldl 

Area  Analysis 

Joint  Integrating  Concepts  (JIC) 
Integrated  Architectures  (IA) 

FNA 

Functional 
Needs  Analysis 


DOTMLPF 

Analysis 


Will  ateriel 
Changes  Solve 
deficiency? 


deas  For 
ateriel 
pproaches 


Will  Integrated 
DOTLPF  changes  solve 
deficiency? 

Recommend  DOTLPF 

cnanges  10  solve  aeTiciency 
(CJCSI  3180) 

o  # 
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FAA  &  FNA  JCIDS  Analysis 


FAA 

•  Joint  Concepts 

•  Threat  &  Environment 

•  Military  Objectives 

\ 

Required  Joint 
Capabilities 


Current  & 
Programmed  Joint 
Capabilities 


FNA 


Required 
Capability  Gaps 


Attributes  of 
Solutions  to  Gap 

I 

MOE 


Strategy-to-Task 


Priority 

Subtasks 

Benefit 

Value 

Legacy 

Program 

1 

Program 

2 

Program 

3 

Program 

4 

1 

Subtasks  12 

55 

40%  I 

20% 

Solution 

10% 

Solution 

2 

Subtasks  21 

46.7 

60% 

35% 

3 

Subtasks  13 

29.4 

Gap  -  No  solution 

4 

Subtasks  9 

14.4 

30%  | 

15% 

43% 

5 

Subtasks  1 

10.3 

90% 

1  No  Major  Gap  -  Existing  DOTMLPF  meets  need  1 

6 

Other  Subtasks 

Subtask  x 

1.2 

100% 

No  Gap 

-  Existing  DOTMLPF  meets  need 

Gap  Analysis 


Attributes  & 
MOE 


Defining  the 
Solution  Set 


Provides  the  Foundation  for  the  Functional 
Solutions  Analysis  (FSA)  &  Ultimately,  the  I  CD 
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Mission  Task  Hierarchy 


Joint  Tasks 


MBT&E  Planning  Process 


•  Four  Basic  Elements: 

-  Mission  analysis  (Critical  Operational  Issue  (COI) 
definition) 

-  System  performance  measures  (attribute 
traceability  to  functions) 

-  Operating  conditions  (test  scenario/environment 
description) 

-  Test  variables  (controlled  and  uncontrolled) 

•  These  items  form  the  basis  for  the  Scope  of 

Test  and  resource  requirement  estimates  that 

are  included  in  the  TEMP 
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Gap  Assessment  Process  Flow 


1.  Analysis  Results 


2.  Subtask  Gap 
Assessment 


3.  Gap  Assessment  Across 
Operational  Scenarios 


4.  Gap  Assessment  at  Task 
Level 


s 

5.  Attributes  for  Gap  Solutions 


6.  Compiled  Gap  Assessment 


Evaluating  the  Mission 


7.  Prioritized  & 
Grouped  Results 

March  2009 


Gap  1:  Conduct  Fire  and  Maneuver 


•  EO  Gap  1  Description:  The  EAF  combat  elements  cannot  move  rapidly 
&  safely  as  a  cohesive  force  while  executing  deep  operational 
maneuver 


•  EAF  must  be  able  to:  Conduct/support  extended  ops  w/  armor 

-  Move  light  armor  by  air:  Employ,  via  air,  the  light  armor  elements  of  EAF  to  achieve 
positional  advantage;  HOnm  in  8  hrs  (TH),  6  hrs  (Obj) 

-  Breach  obstacles,  manmade  and  natural:  Combat  element  must  maneuver  through  or 
around  any  obstruction  designed  or  employed  to  disrupt,  fix,  turn  or  block  movement 
without  delaying  the  force  longer  than  1  hr  (TH),  .5  hr  (Obj). 

-  Protect  the  force  from  the  lethal  effects  of  kinetic  energy  weapons  systems:  Detect  & 
protect  the  force  against  blast,  flame,  thermal,  fragmentation  and  ballistic  effects  by 
equipping  75%  of  the  force  (TH);  equipping  &  training  100%  of  the  force  (Obj). 

-  Provide  Combat  ID:  Attain  an  accurate  characterization  of  detected  objects  -  friend, 
enemy,  neutral  -  in  the  battlespace  by  employing  Active  Recognition  and  Tracking 
Systems  and  Passive  Tracking  Systems  in  100%  of  the  force  (TH  &  Obj). 


•  Characteristics  of  the  Gap 

-  No  capability  to  reposition  light  armor  by  air 

-  Lack  of  mobility  for  vertical  lift  forces 

-  Weight  of  inherent  protection  for  combat  systems  adversely  impacts  EAF  mobility 

-  Unacceptable  limitations  in  the  EAF’s  combat  forces’  ability  to  detect/detonate 
explosive  obstacles 


-  Lack  of  active  recognition  and  active  tracking  systems  for  employment  with  assault,  CS 
or  CSS  elements  of  the  EAF 
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Example  Mission  Task  (COI) 


Issue:  Move  Light  / nfantry.  (Airbome/Air  assault )  via  gmund. 


(Sub-Issue)  The  JLTV  Payload 
Category  B  Vehicle  will 
support... (based  on  unitT/E) 

-  Payload  Characteristics 

•  Transport  9-man  team 

-  P-Spec  Attributes 

-  Performance  Characteristics 

•  Air  Transport 

-  P-Spec  Attributes 

•  Mobility 

-  P-Spec  Attributes 

-  Protection  Characteristics 


(Sub-Issue)  the  JLTV  Payload 
Category  C  Vehicle  will 
support... (based  on  unit  T/E) 

-  Payload  Characteristics 

•  Transport  Unit  Shelters 

-  P-Spec  Attributes 

-  Performance  Characteristics 

•  Air  Transport 

-  P-Spec  Attributes 

•  Mobility 

-  P-Spec  Attributes 

-  Protection  Characteristics 


•  Ballistic  Survivability 

-  P-Spec  Attributes 

•  IA 

-  P-Spec  Attributes 

Suitability  Characteristics 

•  Availability 

-  P-Spec  Attributes 

•  Safety 

-  P-Spec  Attributes 
Evaluating  the  Mission 


•  Ballistic  Survivability 

-  P-Spec  Attributes 

•  IA 

-  P-Spec  Attributes 

Suitability  Characteristics 

•  Availability 

-  P-Spec  Attributes 

•  Safety 

-  P-Spec  Attributes 
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Mission  Evaluation 


£ 

£ 


c 

o 

I 

Q. 


.Q 

o 

a 

t/> 

c 

2 

H 


12 

(0 


3 

CO 


© 


«r 


range 

acceleration 


speed 

range 


braking 


acceleration 


etc 


braking 


etc 


Ballistics 


CBRNE 


etc. 


Ballistics 


CBRNE 


etc. 


CH  47/53 


2  x  I  AT  Cl  30 


CH  47/53 


1  x  I  AT  Cl  30 


Availability 


Safety 


etc. 


Availability 


Safety 


etc. 


Evaluating  the  Mission 


Sub-Configurations  (Varient)  Matrix 
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Mission-Based  COICs 

•  Move  Light  Infantry  (Airborne/Air 
assault)  via  ground 

•  Payload  Category  B  Attributes 

•  Payload  Category  A  Attributes 

(Category  mix  based  on  UnitT/E) 

•  Move  Combat  Support  forces 
via  ground 

•  Etc. 

Mission-Based  ROIs 

•  Add  other  Mission  Tasks  only  as 
necessary  based  on  planned  tests 
(e.g.,  C2) 

Risk  Analysis  for  TD  Phase 

•  Vehicle  capabilities  indicate  potential 
to  meet  COI/ROI 

•  Vehicle  limitations  indicate  risk  area 
to  meeting  COI/ROI 


A 

Task:  What  are  the  lethality  capabilities  and  limitations  of  the  EFSS 
when  performing  suppression  missions? 

OTRR 


Attributes  Result  1  Mission  Measures 
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Conceptual  Evaluation  Process 
(Table  Version) 


Mission  Capability  Assessment 
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Mission  Assessment 


System  Capability  Evaluation  Table 
(Each  Category  by  Vendor) 


Use  Measure  results  to  evaluate  Attributes  and  support  COI 
Evaluation  as  Met,  Partially  Met,  or  Not  Met 
Overall  System  Assessment  based  on  weighted  COI 
"performance" 
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Net 

Ready 


INTEGRATED  SYSTEM  EVALUATION  -  Category  A  by  Vendor 
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Mission  Capability  Evaluation  Table 


•  COIs  assessment  (previous  table)  feeds  Mission  Capability 
Evaluation 

•  Evaluate  COIs  in  the  context  of  supporting  individual  Mission 
Capabilities  (limit  missions  to  most  critical/probable?) 

•  Across  all  participating  variants,  evaluate  impact  on  each 
Mission  Capability  as  Met,  Partially  Met,  or  Not  Met 


MISSIONS 


MSN  1  (Conduct  Mounted  Movement  to  Contact)  (Met,  Partially  Met,  Not  Met) 


Overall 

Variant 

Assessment 

t 

"LT  t  ,t 

C 

General  Purpose 

Inf  Carrier  (Army) 

Infantry  Carrier  (USMC) 

C20TM 

Hvy  Guns  Carrier 

Close  Combat  Wpns  Carrier 

Utility 

Ambulance 

Shelter  Carrier 

Ambulance 

COI 

Move 

Not  Required 
for  this  msn  (NR) 

NR 

NR 

NR 

NR 

NR 

Transport 

Carry 

Payload 

Sustain 

Survive 

Safety 

Net  Ready 
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Summary  (1  of  2) 


•  Significant  analysis  is  conducted  in  the  requirements 
development  process 

-  Mission  Tasks,  Gaps  and  MOEs  identified 

-  Alternatives  selected  based  on  performance  against 
thresholds 

•  Relationship  between  Mission  Tasks  and  System 
Functions  established  in  JCIDS  analysis  is 
maintained  during  the  SE  decomposition 

-  Mission  Profile  analysis  is  key  to  evaluating  Suitability 
characteristics 


•  Test  develops  system  knowledge  over  time 


-  All  phases  of  test  support  evaluation  of  system  “maturity” 

-  Operational  Test  evaluates  the  effect  of  the  System  on  the 
Unit  Mission  performance 
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Summary  (2  of  2) 


•  Fiscal  and  schedule  realities  typically  drive  testing  to 
focus  on  COIs  and  KPPs 

-  System  evaluation  focuses  on  Gap  Missions  and  System 
Functions/Attributes  that  support  mission  effectiveness 

-  Evaluate  Critical  Tasks  and  Issues  to  identify  risk  and  scope 
of  unknown  performance 

•  Potentially,  test  results  would  be  used  to  validate 
early  M&S  assumptions  and  analysis 

•  Did  the  system  deliver  the  expected  capability? 
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QUESTIONS 
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BACKUP 
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Mission  Profile  -  Operational  Context 


MCO 


Secure  LOC 


IrW  -  Arid/Mountain 


Stability  Ops 


COIN  Ops 


Humanitarian 

Relief 


Humanitarian 

Relief 


Stability  Ops 


Operational  Context 

•MPCs  arrive  in  theater  aboard  MPF  shipping  ;  move  ship  to 
shore  at  SPOD  or  via  connector  (e.g.  LCAC) 

•Support  infantry  battalion  with  three  variants: 

•Support  infantry  battalion  across  ROMO 

•  Offensive  Ops:  patrolling,  movement  in  support  of  maneuver, 
urban  ops 

•  Defensive  Ops:  patrolling,  support  by  fire  positions 

•  Stability:  patrolling,  security  ops,  QRF,  checkpoints,  convoy 
security 

w  Evaluating  the  Mission 


Summary 

•MCO  oriented  on  forces  initially;  then  control  of  key  areas  (APODs  / 
SPODs  /  Forward  Bases  /  Key  Cities)  and  routes  between  those  areas 

•Both  IrW  scenarios  oriented  on  control  of  key  areas  /  routes  and 
restoring  host  nation  capability 

•Even  during  MCO,  large  %  of  operations  =  stability  operations 

•Stability  operations  drives  larger  %  of  on-road;  wider  variety  of  mission 
use 
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A 

Roll  Up  Mission  Risk  Tables 


•  Assess  the  risk  consequence  and 
probability  to  effectively  support  the 
designated  missions  capabilities. 


By  Category 
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Structuring  T&E  for 
of  Complex  Systems  Capabilities 
and  Exploration  of  Emergent  Behaviors 


Agenda 


SYSTEMS  THEORY  RELEVANT  TO  T&E 

-  Complex  Systems 

-  Holism 

-  Emergence 

STRUCTURING  T&E  FOR  VALIDATION  OF  COMPLEX 
SYSTEMS 

-  Complex  System  V&V 

-  Complex  System  T&E 

-  Considerations  and  Recommendations 

•  Mission  and  Functional  Analysis  Based  Test  Planning 

•  Risk  Prioritized  Integrated  Testing  across  the  Development  Life-Cycle 

•  Design  of  Experiments  Based  Test  Planning  and  Analysis 


CONCLUSION 


Systems  Engineering 

The  way  it  really  is? 


How  the 
user  described  it 


How 

the  requirement 
was  understood 


How  the 
contractor 
designed  it 


How  the 
programmer 
wrote  it 


How  the 
PM/sponsor 
described  it 


How  the  project 

What  was 

How  the  1 

1  How  the  helpdesk 

What  the  user 

was  documented 

actually  installed 

Government  was 
billed  | 

supported  it 

really  needed 

Original  Source  Unknown 
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SYSTEMS  THEORY  RELEVANT  TO  T&E 

-  Complex  Systems 

-  Holism 

-  Emergence 


Complex  Systems 


Holistic,  hierarchal,  transient,  emergent,  non-deterministic,  large 
number  of  elements  and  states,  etc... 

Detail  complexity — hierarchical  relationships  dominate;  complex  in  size 
and  scope  but  straight  forward  causal  relationships;  somewhat  predictable; 

risks  in  contributing  parts  (Calvano  &  John,  2004) 

Dynamic  complexity — lateral  interfaces  dominate;  complex  integration 
and  behaviors;  unpredictable;  system  risks  dominate;  causal  relationship! 

difficult  if  not  impossible  to  determine  (Calvano  &  John,  2004) 


a,  Weakiy-integrated  system 


b.  Highly-integrated  system 


Figure  from  Calvano,  C.  &  John,  P.  (2004).  “Systems  Engineering  in  an  Age  of  Complexity,”  Systems  Engineering,  7(1),  p.  27. 


•  Very  Complex  System;  also  possibly  a  System  of  Systems 

•  Dynamic  Complexity  -  many  parts,  high  degree  of  coupling  and  interaction  across 
a  large  number  of  complex  interfaces;  integration  challenge! 

•  Diverse,  separated,  component  systems;  transient  states  &  deterministic  behavior 

•  Variability  and  unpredictability  of  human  behaviors  and  interaction  w/  system 

•  Emergent  behaviors  from  coupling  of  sensors,  weapons,  &  employment  methods 
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Complex  Systems  Example: 

Naval  Surface  Fire  Support 
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Holism 


“A  system  has  holistic  properties  possessed  by  none  of  its  parts. 

Each  of  the  system  parts  has  properties  not  possessed  by  the  system 


as  a  whole.” 


Clemson  (1984,  p.  201) 


Holism  emphasizes  whole  over  parts;  organizational  level  analysis 
ensuring  elements  function  together  to  serve  the  purpose  of  the 

system.  Jackson (2006,  p.  650) 

Reductionism  serves  to  help  in  building  up  a  system  design,  but  a 
holistic  view  is  needed  in  order  to  evaluate  complex  systems. 


Figure  from  Old  Dominion  University  ENMA  640  and  ENMA  715  Course  Presentations 


Emergence 


A  systems  principle  that  whole  entities  exhibit  properties  that  are 
meaningful  only  when  attributed  to  the  whole,  not  its  parts  -  e.g.  the 
pungent  smell  of  ammonia,  which  comes  from  the  properties  of  the 
molecule,  not  those  of  the  constituent  nitrogen  and  hydrogen  atoms. 


(Hitchens,  2003) 


Generally  accepted  aspects  of  emergence  (see  backup  slides  for  references  from 

literature): 

•  Unpredictable  /  unexpected 

•  Derives  from  integration,  interfaces,  and  interaction  of  the  systems’ 
elements 

•  Not  present  in  the  systems’  elements  nor  can  it  be  predicted  by  evaluating 
each  of  those  -  although  can  influence  them 


Figure  from  Old  Dominion  University  ENMA  640  and  ENMA  715  Course  Presentations 
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STRUCTURING  T&E  FOR  VALIDATION 
OF  COMPLEX  SYSTEMS 

-  Complex  System  V&V 

-  Complex  System  T&E 

-  Considerations  and  Recommendations 


Verification  &  Validation 


Verification:  Confirmation  by  examination  and  provisions  of  objective 

evidence,  that  specified  requirements  have  been  fulfilled.  (lEEEStd  1012,  1998,  p.71) 

Did  you  build  what  you  said  you  were  going  to  build? 

Validation:  Confirmation  by  examination  and  provisions  of  objective 
evidence  that  the  particular  requirements  for  a  specific  intended  use  are 

fulfilled.  (IEEE  Std  1012,  1998,  p.71) 

Did  you  build  what  the  user  needed? 

V&V  is  a  broad  set  of  activities  for  software  or  whole  systems  applied 
across  the  life-cycle,  supporting  a  variety  of  activities  from  assessing 
technical  alternatives  in  conceptual  design  to  ensuring  the  as-built 
system  meets  its  specifications  and  the  user’s  need. 
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What  encompasses  a  user’s  needs? 
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Figure  from  DoD  Acquisition  Guidebook  (2006).  Washington,  DC:  OSD  AT&L.  p.  141 . 


Test  &  Evaluation 


Test  =  procedure  to  measure  performance  under  various  conditions. 


(Parker, 


1994) 


Testing  =  linchpin,  intelligence/feedback  loop  for  systems  engineering, 

(Laskey,  1999,  p.  6) 


T&E  at  its  best  =  goal  to  provide  necessary  info  to  ensure  quality. 
At  its  worst  =  political  game  where  programs  fight  for  survival  and 

money.  (Laskey, 1999,  p.  6) 


•T&E  ~  experiment  &  theory;  predict  outcome,  validate  points  of  doubt 

(Goode  &  Machol  (1957,  p.  509) 

-Unfortunately  complex  systems  are  often  non-deterministic  if  not  completely 
unpredictable 


Focus  of  T&E 


“How  well  did  the  system  actually  perform,  and  did  it 
accomplish  its  mission  objective?”  - 

-  Mission  accomplishment  not  just  specs 

-  System  effectiveness  +  Support  system  capability  (suitability) 


“Does  the  system  meet  all  of  the  requirements  as  covered 
through  the  specified  technical  performance  measures?” 

•  Technical  verification;  developmental  testing 


•  “Does  the  system  meet  all  [user]  requirements?” 

•  Verification  and  validation? 

•  Could  a  system  meet  the  user’s  requirements  as  stated  and 
yet  be  not  operationally  effective/not  suitable? 

Blanchard  (2008,  p.  113) 
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Complex  Systems  Engineering  +  V&V  /  T&E 
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Figure  adapted  from  Blanchard,  S.  &  Fabrycky,  W.  J.  (2006J. 
Systems  Engineering  and  Analysis.  (4th  ed.)  Upper  Saddle 
River,  NJ:  Prentice-Flail. 


System  Engineering  Requirements 
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System 
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Improvemeri 

Get  it  riaht  early! 

Importance  of  Conceptual  Design 
Get  testers  involved  (OT/DT/LFT) 
Understand  mission  environment 
Understand  what  constitutes 
operational  effectiveness  &  suitability; 
what  user  really  needs! 

Test  early  &  often  w/  honest 
assessment  of  risk  to  sustained 
mission  capability 


Lifecycle  phase 


Source:  Ford,  R.  B.  &  Barkan,  P.  (1995)  “Beyond  Parameter  Design  --A  Methodology  Addressing 
Product  Robustness  at  the  Concept  Formation  Stage”,  Design  for  Manufacturability,  1995  Concurrent 
Engineering  and  Design  Manufacturing  Integration,  ASME  DE-Vol.  81,  J.R.  Behun  (Ed.),  National 
Design  Engineering  Conference,  Chicago,  IL,  March  13-16. 


Program  Context  and  Systems  Evaluation 


Implications  for  complex  system  T&E 

•  Understand  mission  environment  and  ensure  testing  explores  those  boundaries 

•  Evaluation  of  system/program  performed  in  context  of  the  program  &  many  external  and 
internal  influences  and  drivers 

•  These  also  affect  capabilities  and  limitations  of  T&E  itself 

Figure  adapted  from  Blanchard,  S.  &  Fabrycky,  W.  J.  (2006,).  Systems  Engineering  and  Analysis.  (4th  ed.)  Upper  Saddle  River,  NJ:  Prentice-Hall. 
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mplications  for  complex  system  T&E 

ultiple,  concurrent  missions  evaluated  in  complex  scenarios 
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Total  Platform/System  Mission  Context 
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Mission  Based  Testing 

T&E  Framework  Toolset  Architecture 


System  CDD  Capabilities  /  Requirements 
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Other  References  (System  CONOPS,  TTP,  Specs) 
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AVW  Technologies,  Inc. 
Proprietary 


Describe  discrete  tasks  to  perform 
system  missions  in  user’s  language 

Define  conditions  affecting  task 
outcome 

Extract  attributes  &  measures  from 
requirements  &  other  references; 
correlate  to  tasks 

Assign  test  methods  &  data 
requirements  to  measures 

Group  tasks  w/  conditions  & 
measures  into  testable/meaningful 
vignettes 

Repeat  vignettes  to  account  for 
variability  due  to  conditions  and 
settings 

Test  vignettes  grouped  into 
scenarios  run  during  test  events 
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Integrated  Testing 


Planning  and  development  of  individual  test  objectives; 
Coordinated  integration  of  objectives  over  life-cycle 

Leveraged  tests  and  data  but  independent  evaluation 
Potential  cost  savings  +  better  risk  reduction;  may  reduce 
but  cannot  eliminate  IOT&E 

Requires  buy-in,  strong  T&E  WIPT,  rigorous  and  early 
planning  and  cooperations 


Integrated  Testing 


JointExpAJC 


Separate 
Evaluations 
&  Reports 

Joffki:  JC-TBs 


AVW 

Technologies, 
Inc.  Proprietary 


System  Disposal 

T “—“I'  '  ^  °T’  ^  ^  * 

Program  Conception 
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DT 


Integrated  Testing 

OT 


Test  to  specifications 
Limited  test  environment 
Focused  on  a  specific  set  of  criteria. 
Test  threshold  values  not  capability 
Critical  technical  parameters 

Integration  testing  designed  around 
min  performance  criteria  and 
interface  specs. 


Operational  environment  &  threat  with 
end  users  &  support 

End-to-end  mission  perf.  &  support 

Production  representative;  system/ 
family  of  systems 

Test  overall  capability  of  an  item  to 
meet  user’s  mission  needs  and  value 
added  for  mission  accomplishment. 

Test  the  limitations  and  capabilities  of 
an  item  so  that: 


May  not  address  all  threats  or 
missions. 

CT  adds  contractual  issues 


Employ  and  assess  doctrine/TTP 

Independent  IOT&E  &  LFT&E 
mandates  (Title  X) 


THIS  MUST  TRANSFORM  INTO  A 
CONTINUUM  OF  TESTING 

•  Increasing  fidelity  of  technical  and 
operational  assessments 

•  Cooperating  organizations 

•  Reduced  budget  and  timeline  ? 

•  Team/IPT  structure  not  competitive 
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f  Materiel 

Technology 

Engineering  and 

Production  & 

Operations  & 

Solution 

Analysis 

Development 

Manufacturing  Development 

Deployment 

Support 

Systems  Engineering  +  T&E 
within  the  Acquisition  Cycle 


FOC 


THE  SYSTEMS  ENGINEERING  PROCESS 


THE  SYSTEMS  ENGINEERING  PROCESS 


THE  SYSTEMS  ENGINEERING  PROCESS 

"2 


Funtiinal  Analysis/ 
Allocation 


ViMlicitioa  ^_0hIj»Lm  ^ 


Proem 

\| 


THE  SYSTEMS  ENGINEERING  PROCESS 


Funtional  Analysis/ 
Allocation 


t  Oesign  Lapp  i 


|\ 


Concept  V&V 

Prototype  Testing/Exp/Analysis 

Component  Testing 
Functional  Arch  V&V 

Subsystem  Testing  System  DT  &  OT 

Physical  Arch  V&V  Config  Audits 

Follow  on  OT 
Upgrade/Re-arch  V&V 

- - - 

MORE  THAN  TESTING  ...  CONTINUOUS  EVALUATION 


RISK  MANAGEMENT.. 


MISSION  CAPABILITY  DELIVERY 


INTEGRATED  T&E  &  SYSTEMS  ENGINEERING 


Ability  to  influence 
system  design 


System  maturity  & 
design/  upgrade  cost 


Design  of  Experiments 

Process  Overview 


Project  description  and  process  decomposition 

•  Problem  statement  and  objective  of  experiment  (test) 

•  Response  variables,  and  potential  causal  variables  -  Ishikawa  fish  bone. 

Plan  test  matrix 

•  Determine  constraints,  prioritize  factors,  and  select  statistical  design  (2Kvs.  3Kvs.  mixed,  Taguchi 

vs.  classical  arrays,  full  vs.  fractional,  non-linear  effects?,  replications?,  blocking?) 

•  Write  the  test  plan  with  sample  matrices,  profiles,  and  sample  output;  run  sample  analysis. 

Produce  observations 

•  Random  run  order  &  blocked  against  unknown  effects 

•  Block  runs  to  guard  against  uncontrollable  unknown  effects  as  needed. 

Ponder  the  results 

•  Analyze  and  project  data;  draw  conclusions,  redesign  test  as  necessary  and  assess  results. 

•  Perform  “salvo  testing”  (test-analyze-test);  screen  large  #  of  factors  then  model 


Process 


Plan 


Ponder 
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Reft  Hand 
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Design  of  Experiments 

Test  Matrix  Development 


Partial  or  full  matrix  of  varying  settings  for  the 
factors  (usually  2  or  sometimes  3  level) 

Perform  larger  matrices  in  increments,  eliminating 
factors  that  are  shown  to  be  non-factors  through 
analysis 

Goal  is  to  determine  cause  of  variability  in  output 
based  on  input  factors 


Output  measured  at 
each  point 


Run  4 


Run  8 


Controllable  Factors 

^  ^  ^  Outnuts  ( v ) 

Inputs  (x) 

- ► 

k 

Environmen 

f  t  t 

tal  /Uncontrollable  Factors 

Run  6 


Factor  (A) 


Run 

Setting 

A 

6 

c 

1 

Factorial  point 

-1 

-1 

-1 

2 

Factorial  point 

-1 

-1 

+1 

3 

Factorial  point 

-1 

+1 

-1 

4 

Factorial  point 

-1 

+1 

+1 

5 

Factorial  point 

+1 

-1 

-1 

6 

Factorial  point 

+1 

-1 

+1 

7 

Factorial  point 

+1 

+1 

-1 

8 

Factorial  point 

+1 

+1 

+1 

9 

Center  point  1 

0 

0 

0 

10 

Center  point  2 

0 

0 

0 

Adapted  from  USAF  53rd  T&E  Wing  DOE  Training  Materials 


Design  of  Experiments 

Data  Analysis 


Sample  data  analysis:  produces 
regression  model,  predictions, 
response  surface/curves  as 
shown. 

Statistical  analysis  of  the  MOEs 

Exploration  of  mission 
performance  variability  across 
driving  conditions 


-1.2  -1.0  -0.8  -0.6  -0.4  -0.2 


0.2  0.4  0.6  0.8  1.0  1.2 


Tribble,  J.  E.  (2008)  “Applying  Design  of  Experiments  Methodology  to  Sortie  Generation  Rate  T&E”  Proceedings  of  the  24th  Annual  NDIA  Test  & 
Evaluation  Conference,  February  25-28,  2008.  p.  20.  Downloaded  from:  http://www.dtic.mil/ndia/2008test/aqenda.pdf.  Note:  derived  from  actual  test 
data  using  response  surface  analysis  methods  in  Montgomery  (2005),  Schmidt  &  Launsby  (2005). 
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Design  of  Experiments 

Benefits 

Better  way  to  design  and  test  complex  systems 
Systematically  explores  system  performance,  effectiveness,  and 
suitability  -  breadth  and  depth  of  testing  across  the  performance 
envelope 

Challenge  assumptions  and  demonstrate  real  performance  across  the 
expected  environment 

Better  justification  for  sample  sizes;  potentially  significant  reduction 
from  case  or  one  factor  at  a  time  testing 
Better  linkage  between  M&S  and  live  test 
DOE  works  well  in  Mission  Based  Testing/Integrated  Testing  and 
relies  on  proper  task  derivation  and  attributes  and  conditions  selection 
DOE  can  improve  testing  across  all  programs  -  it  is  simply  smarter 
testing! 
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Conclusion 


•  Complex  Systems 

-Complex  Systems-  detail  vs.  dynamic  complexity 
-Holism 
-Emergence 

•  Structuring  T&E  to  Validate  Complex  Systems 

-Understand  program  context  and  system  mission  context 
-Test  early,  test  often... 

-Mission  Based  Test  Design 
-Integrated  Testing 
•CT  +  DT  +  OT  +  LFT  +  ... 

•  Integrated  testing  and  data  collection 

•  But. ..Independent  evaluation 
•T&E  as  part  of  SE  life-cycle 

•  Design  of  Experiments 

•  Questions 
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THREAT/TARGET  SYSTEMS 

T&E 


Gregg  Van  Splinter 


DISTRIBUTION  STATEMENT  A.  Approved  for  public  release; 
distribution  is  unlimited. 


Mission 
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4REAT/TARG ET  SYSTEMS  DEPARTME  hi 


OUR  MISSION  IS  TO  EMULATE  THREATS  FOR  WEAPONS 
AND  EW  SYSTEMS,  TEST  AND  EVALUATION  AND  TO 
SUPPORT  EXPERIMENTATION  AND  FLEET  TRAINING 


LAND/AIRVSEA 


KINEMATICS: 

SPEED  VS.  ALTITUDE 
MANEUVERABILITY 

SIGNATURES: 
tNFRARED 
RADAR 
t m  VISUAL 

COUNTERMEASURES. 
ELECTRONIC 
INFRARED 

RF  EMISSIONS 


UNDERSTANDING  THE  CftlTIGAJ.  MASS 
G  0n  erici'S  urro  gateiVa!  idated/Rtplica^Acfual 
(Cost  Effective  Fidelity  for  T  3.  E) 
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capabilities 
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Target  Systems  Engineering 

Target  systems  technology 
development,  acquisition  support, 
systems  integration,  operations 
engineering,  Els,  LECs 


Atlantic/Pacific  Target  & 
Marine  Operations 

World  wide  surface  and  airborne  target 
and  marine  operational  services  includin 
unique  mods  and  augmentation 


Threat/Target  Systems  Management 


Provides  entry  point  for  Sponsors  to 
form  integrated  project  offices,  and 
externally  directed  project  offices 


Airborne  Threat  Simulation 

Design  development  and  operational 
support  of  electronic  attack  and 
radar  airborne  threat  emitter  systems 


Combat  Environment  Threat 
Simulation 

Development  and  technical  support 
of  EO/IR/MW/UV/Laser/C4l  threat 
simulator  systems 
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TTSD  Operating  Sites 
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Pax  River 


China  Lake 


Wallops 


Point  Mugu  / 
Port  Hueneme 


PMRF 


SCORE 


Yuma 


WSMR/Holloman 


Norfolk/Dam  Neck 
MCAS  Cherry  Point 

Eglin/Tyndall 


Key  West 


•  CFA  Okinawa 


"k  Department  Operating  Activities 
•  Sites  TTSD  Deploys  To  Regularly 
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Target  Systems  Division 
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PROVIDES  THE  PEOPLE, 

FACILITIES,  AND  EQUIPMENT 
NECESSARY  TO: 

■  EXECUTE  TARGET  SYSTEM 
TECHNOLOGY  DEVELOPMENT 

■  SYSTEMS  ACQUISITION 

■  SYSTEMS  INTEGRATION 

■  IN-SERVICE  OPERATIONAL 
ENGINEERING 

■  TARGET/DECOY  SYSTEMS  USE  FOR 
TEST  AND  EVALUATION  OF  NAVY 
ACQUISITION  PROGRAMS,  FLEET 
OPERATIONS,  AND  TRAINING 

■  TASKS  ASSOCIATED  WITH  THE  |H  « 
PREPARATION  OF  TEST 
INVENTORY  TARGETS  FOR 
SPECIFIC  TEST  PURPOSES 
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Target  Systems  Products 
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New  Target  System  Acquisition  Support  ■  Scoring  System  Engineering 


■  Target  Free  Flight  Clearance  ■  Target  Test  Planning 


■  Modeling  &  Simulation  Laboratory  ■  Target  Engineering  Investigation 
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Aerial  Targets 


Supersonic 


AQM-37C 


GQM-163A 


Subsonic 


BQM-34S 


LI 


Decoys 


TALD 


ITALD 


MSST 
(IOC  FY13) 


Developmental 


SSAT  (IOC  FY12-13) 


ATALD 

(Currently  unfunded) 


N  AV  ^AIR 


■  Provide  seaborne  target  services  to  the 
Fleet  ,DoD,  and  Foreign  Military  Customers 
in  support  of  weapon  system  T&E  and  Fleet 
Training 

■  US  Navy  Cognizant  Field  Activity  for 
Seaborne  Targets  and  augmentation 
systems 

-  DEVELOPMENT 

■  T&E 

■  IN  SERVICE  ENGINEERING/DEPOT 

-  INTEGRATED  LOGISTICS  SUPPORT 

-  SUPPORT  TO  OTHER  OPERATING 
ACTIVITIES 

■  Tri-Service  Lead  for  development  and 
operation  of  Seaborne  Targets 
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Fast  Attack  Craft  Target  (FACT) 
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■  Requirement 

■  Threat-representative, 
open-ocean 
performance  craft 

■  50+  feet 

■  50+  Knots  -  Sea 
State  2 

■  Capable  of  hi-speed 
towing 


■  Development 


■  Two  prototypes  with 
different  hull  designs 
and  propulsion 
systems  were 
developed  and 
evaluated  under 
remote  control 

■  One  design  was 
determined  to  be 
more  threat- 
representative  and  to 
have  superior 
handling  qualities 
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Fast  Attack  Craft  Target  (FACT) 


FACT  is  now  in  production 
and  is  operational 
at  NAWCWD 


Seaborne  Powered  Targets 
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■  i  k 
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Powered  Surface  Targets 
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High  Speed  Maneuvering 
Surface  Target  (HSMST) 


Ship  Deployable  Surface  Target 
(SDST) 


POWERED 

SURFACE 

TARGETS 

POWERED 
SURFACE 
TARGETS  - 
UNIQUE 
HULLS 

UNMANNED 

SURFACE 

VESSELS 


Coastal  Patrol  Interdiction  Craft  (CPIC) 


Unmanned  Surface  Sea 
Vessel  (USSV) 


Mobile  Ship  Target  (MST) 


Aerial  Target  Launch  Ship 
(ATLS) 
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Non-Po  we  red  Surface  Targets 
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Improved  Surface  Tow 
Target  (ISTT) 


Low  Cost  Modular  Target 
(LCMT) 


High  Speed  Anti- 
Radiation  Missile 
(HARM)  Barge 


Low  Cost  Tow  Target 
(LCTT) 


Trimaran 
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Target  Operations  Division 


■  PROVIDES  THE  RESOURCES  TO  PROVIDE 
TARGET  OPERATIONAL  SERVICES  FOR  WEAPON 
SYSTEM  AND  TARGET  DEVELOPMENT,  TEST  AND 
EVALUATION,  FLEET  OPERATIONS,  AND  FLEET 
TRAINING 


Target  Ops  FY07/FY08 
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-  EAST  COAST: 

-  562  SURFACE  EVENTS  (2,700+  MISSION 
HOURS)(FY07) 

■  875  SURFACE  EVENTS  (FY08  NORFOLK) 

-  264  SURFACE  EVENTS  (FY08  PAX  RIVER) 

■  46  AERIAL  TARGET  LAUNCHES  (FEB  08) 

-  WEST  COAST 

■  945  SURFACE  EVENTS  (FY07) 

-  151  AERIAL  TARGET  LAUNCHES  (FY07) 

-  -892  SURFACE  EVENTS  TO  DATE  (FY08) 

■  -168  AERIAL  TARGET  LAUNCHES  (FY08) 


Atlantic  Targets  and  Marine 
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Operations  Division 


ON-SCENE  RANGE  SAFETY 
SURVEILLANCE  AND  CLEARANCE 
OF  THE  CHESAPEAKE  BAY  TEST 
RANGE,  THE  ATLANTIC  WARNING 
AREAS  AND  OTHER  OPEN  WATER 
AREAS 

SURFACE  (STANDARD  NAVY 
TARGETS  AND  FULL  SCALE 
TARGET  HULKS),  LAND,  AND 
AERIAL  TARGETS  FOR  TEST  AND 
TRAINING  OPERATIONS 

DESIGNS  AND  FABRICATES 
PROTOTYPE  TARGETS  TO  MEET 
UNIQUE  CUSTOMER 
SPECIFICATIONS 


OPERATES  AND  MAINTAINS 
MARINE  SURFACE  VESSELS  TO 
SUPPORT  TESTING  AND 
TRAINING 

PROVIDES  UNDERWATER  AND 
LAND  TEST  ARTICLE  RECOVERY 
OPERATIONS 
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Support  Vessels 


■  Navy  Mklll  65ft  Patrol  Boat  777 

B  Navy  Transporter  and  Retriever  (100  ft) 

■  Hurricane  23ft  Rib  Boat 
B  Fountain 
B  Navy  Prince 
-  NAVAIR-38  (Key  West) 
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Support  Vessels 
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Weapons  Effect  Testing 
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1 

l 

REAT/TARG ET  SYSTEMS  DEPARTMENT 

LPD-19  Shock  Test 
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1 

l 

REAT/TARG ET  SYSTEMS  DEPARTMENT 

Ex-USS  Oriskany  Reefing 


N  AV  ^AIR 


Mobile  Launch  Platform 
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BQM-74  Sea  Launch  Capability 
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4REAT/TARGET  SYSTEMS  DEPARTMEK 
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Airborne  Threat  Simulation 

Division 


urar fighter  tv  tin  fiignly 
representative ,  validated,  *— 
timely,  and  affordable  threat 
electron  i c-combat  enviro  nmen  ts 
in  ivhich  to  test  and  train,  " 


LI  KTLi 
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Electronic  Attack  and 
Active  Emitter 


■  Electronic  Attack  (EA) 
Simulators 

B  Simulation  of  jammer  systems 
associated  with  both  manned 
aircraft  and  cruise  missiles 

■  Threat  Radar  Simulators 


USAF  Photo 


Transmitted  Signal  Simulators 
(Stimulators) 

■  Manned  Aircraft 

■  Anti-Ship  Cruise  Missiles 

■  Ground  Based  SAM,  AAA 
Full-up  Seeker  Simulators 

■  Anti-Ship  Cruise  Missiles 
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A  TSO  Airborne  Electronic 

Attack 
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■  Weapons  System 
Development 


Airborne  Electronic  Attack 
AN/ALQ-167(V) 
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■  AN/ALQ-1 67(V)  is  an 
advanced  airborne 
electronic  attack 
simulator  pod  developed 
for  weapon  system  T&E 
and  Fleet  training 

■  The  "Have  All”  system 
can  replicate  several 
families  of  current 
and  projected  threat 
jammer  systems 

■  The  system  is  based 
on  the  AN/ULQ-21  (V) 
countermeasures  set 
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Airborne  Radar  Simulation 


MAX/ A  1  D 

1,  1 

AN/AST-9(V)  is  an  advanced 
supersonic  airborne  radar 
simulator  pod  developed  for 
weapon  system  T&E  and  Fleet 
training. 

The  configurable  AST-9 
simulates  airborne  radar  and 
anti-ship  cruise  missile  systems 
using  high  power  traveling  wave 
tube  and  magnetron 
transmitters. 
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A  TSO  FY08  Operations 

Supported 


R 


ATSO  FY-08  Monthly  Operational  Support  Snapshot 
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Ground/Emitter  Lite 


>  Low/High  Power  Transmitter 
Options: 

Emitter  Lite  (DPT-2B) 

Emitter  Lite  (UPT-2A) 
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Advanced  Ground  Target  Threat 


Plastic  Target 

-  Realistic  Shape,  Size,  RCS,  Visual 


Duaj  Emitters  Track_  and  Search 

Realistic  Emitter  Waveforms 


MO-MDH 


Design  Concept 

-  One  Integrated  Multi-Spectral 

System 
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Combat  Environment  Simulation 


Division 
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THREAT  SIMULATORS-RF,  IR,  UV  FOR  T&E  RANGES 
SUCH  AS  ECR  AT  CHINA  LAKE. 

-  DEVELOPMENT 

-  ACQUISITION 

-  T&E 

-  VALIDATION  DOCUMENTATION 

ELECTRONIC  WARFARE  SYSTEMS  FOR  THE  MAJOR 
DON  AIRCREW  TACTICAL  TRAINING  RANGES. 

•  DEVELOPMENT 

•  ACQUISITION 

•  INTEGRATION 

•  UPGRADES 

NAVY  THREAT  /  SIMULATOR  VALIDATION  PROGRAM 
COORDINATION 

•  INDEPENDENT  OF  DEVEL  OFFICES/JOINT  VAL  MEMBER 

•  DEVELOP  AND  MAINTAIN  VALIDATION  PROCEDURES 

•  REVIEW  VALIDATION  REPORTS 

•  MAINTAIN  DATABASE  AND  SCHEDULES 
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CESD  Test  And  Evaluation 
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■  Open  Air  Range  Threat  Simulation 

-  RF,  MMW,  IR,  EO,  UV 

■  Joint  Asymmetric  Warfare  Systems  (JAWS) 

B  DT  /  OT  /  TAC  D&E  Testing,  Training,  and  Tactics 

■  IR  Simulator  and  Target  Array  (ISTAR) 

■  Advanced  Threat  Simulator  System  (ATSS) 

■  Sustainment  Management  Office  for  GPS 
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JAWS’  Unique  Support  Cycle 
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Provides  ASE  and  Training 
Requirements  for... 


◄ 

■  Combat  Environment  Simulation 


Warfighter  Fleet  Support 


■  IR/EO/UV/Laser  &  RF  Threat  Simulators  -  ASE  Readiness  Flightline  Checks 


■  OPFOR  Low  Cost  Stimulators 


MANPAD  Tactics  and  Training 


■  Rapid  Response  QRC  Prototyping  _  Warfighter  Response  Center 

■  HFI  and  Threat  Signature  Collection  _  UUNg  jcaT,  SAFIRE  SMEs 


Develops  Threats 
and  Scientific 
Expertise  for... 


JAWS 


Provides  vetted  Data, 
Tactics,  SOPs,  and 
Crisis  Response  for... 


DT  /  OT  /  TAC  D&E  Test  &  Evaluation 

-  DT  /  OT  Testing  at  NAWC-WD  (VX-9) 

-  TAC  D&E  at  NSAWC  and  MAWTS-1 

-  OPFOR  Training  at  MAWTS-1,  NSAWC 

-  M  &  S  for  TSPIL  and  H-1  WSSA 


Tactics  Development  &  Training 

■  Tactics  Optimization  &  Training 

■  Mission  Planning  and  Analysis 

■  NATOPS  /  NATEC  Documentation 
-  Support  ADT&L  at  MAWTS-1 


- ► 

Provides  IRCM  &  ASE 
Effectiveness  Data  for... 


12  Feb  2007 
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IR  Stimulator  and  Target  Array 
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■  Tactical  Aircraft  Programs  Are  Developing  Advanced  Systems 
That  Detect,  Target,  and/or  Counter  Threats  in  the  Infrared  and 
Electro-  Optical  Spectra 

■  Open-Air,  End  to  End,  Closed  Loop  Test  Capabilities  Required 

■  CES  Developed  End  to  End  IRCM  Test  and  Evaluation  Capabilities 

-  IR,  UV,  and  Laser  Simulation  (MWS  &  DIRCM  Tracker) 

■  Jam-Beam  Radiometry 

■  Actual  Passive  Threat  Missile  Seekers 
(Guidance  Electronics) 

■  Missile  Fly-out  Models  (Miss  Distance) 

■  Real-Time  Atmospherics  (Environmental  Effects) 

■  Real-Time  IR  Scenes  (Target,  CM,  Environment) 

■  Missile  Closure  /  End-Game  Evaluation 

■  TSPI  and  Multi-Spectral  Ground  Truth 
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Open  Air  Range  Threat  Simulation 


iR/UV  Test  Capability 
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Mallinas 
(UV  Plume 
Simulator) 


ISTAR 

(IR  Plume  Simulator) 


Atmospherics  Ground  Truth 


FFM 

(Seeker  Test  Van) 
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OAR  SPIL 

Open  Air  Range 
Signal  Processor  in  the  Loop 


Data  Processing 


EW  Live  Threat  Training 
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■  CESD  is  the  major  field  activity  for 
acquisition  and  development  of  the 
RED  Force  Surface  to  Air  /  IADS 
threats  for  DON  major  TTR’s. 


AN/GPQ-11 
Fallon,  NV 
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■  Target  Mold 
Prototype  & 
Manufacture 

■  3D,  2D,  cold  air 
inflatable 

■  Copper  cladded 

■  Trailer  mounted 

■  Active  Emitters 
Integrated 
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Examples  of  Current/Future 
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■  Seaborne  target  swarm  capability 
for  Fleet  training 

■  VBSS  support  to  Navy,  Coast 
Guard,  Home  Land  Security 


■  Advanced  Ground  Target  Threat 
System/Land  Target  Development 

■  Acquisition  support  and  T&E  of 
SSAT  (BQM-74E  replacement) 

■  Acquisition  support  and  T&E  of 
MSST 


■  Future  testing  of  Hi  Diver  variant 
of  SSST 


Threat/Target  Systems 


Department  Summai 

^ _ g. 

N  AV^A  1  R  1 

II 

■  T/TSD  has  tremendous  operations  and  engineering 
capability 

■  We  operate  wherever  the  customer  needs  us 
B  We  are  committed  to  constant  improvement 

■  The  Military  Value  of  our  products  and  services  is 
recognized  by  our  Navy,  DoD  and  FMS  Customers 

■  We  see  many  opportunities 

■  Target  Operations  World  Wide 

■  New  Target  Developments 

■  Homeland  Security  exercise  scenario  support 
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QUESTIONS? 


N  AV  ^AIR 
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America’s  Incredible,  Shrinking,  Aging,  Less  Ready, 
More  Expensive  Armed  Forces 

Winslow  T.  Wheeler 
Center  for  Defense  Information 
Wheeler@CDI.org 


The  talent,  judgment,  and  insight  collected  in  this,  booh  are  phenomenal.  Over  the 
lad  generation,  the  authors  have  been  more  ri^it,  mare  often,  about  mare  issues 
of  crucial  importance  to  American  security  than  any  other  croup  I  can  thin k  of.  It  is 
a  tremendous  benefit  to  have  their  views  collected  in  one  place  and  concentrated 
on  the  next  big  choices  facing  a  new  administration.  This  really  is  a  book  that  every 
serious-minded  citiien  should  read. 

James  Fallow  i,  National  Correspondent.  The  Atlantic  Monthly 


T  AMERICA’S 

A  DEFENSE  MELTIOWH 


9  "7flD8D4*7  &9  J  Hr 
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Mr.  President,  you  know  the  national  economy  is-  in  a  shambles.  But  you  probably 
don't  know  the  national  defense  is,  too.  according  to  a  must  read  book,  "America's 
Defense  Meltdown." 

George  C.  Wilson,  former  chief  military  correspondent,  The  kVashingtojv  Post 


The  next  big  American  implosion,  comparable  to  the  meltdown  of  our  financial  system, 
is  likely  ta  occur  in  the  Pentagon  and  the  military-indm-trial  complex.  For  decades 
we  have  been  going  deeply  into  debt  buying  weapons  that  we  don't  need  nnd  don't 
work.  Winslow  Wheeler  and  his  uniformed  nnd  civilian  defense  experts  arethe  best 
informed  analysts  of  military  pre  pa  redness  and  procurement  we  have.  They  paint  a 
picture  of  totally  broken  and  dysfunctiona  I  military  institutionsthat  are  simply  waiting 
for  the  slightest  push  to  send  them  over  the  edge.  This  bookshould  be  indispensable 
reading  for  the  next  president  and  his  defense  advisers. 

Chalmers  Johnson,  author  of  Sloivback  (2CC0],  The  Sorrotvs  of  Empire  (2C04), 
and  Nemesis  The  Last  Days  of  the  .4mencaj»  Republic  (20  04) 


These  authors  have  been  forcing  defense  analysts  to  rethink  ther  positions  for  years. 
While  you  may  not  agree  with  all  they  say,  I  can  guarantee  they  will  force  you  to 
reconsider  how  America  will  insure  its  future. 

Col.  Thcmas  X  Hanimes(U.S.  Marine  Corps,  ret.), 
author  of  The  Singaivcf  The  Stone 


W inflow  T.  W  heel  er  is  Di  rector  of  the  Straus-  M  il  ita  ry  Reform  Project  at  the 
Center  for  Defense  Information. 
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Pentagon  Reform  for  President  Obama  and  the  New  Congress 


THE  PHNTAgq.  - 


Thirteen  nou-partisin  Pentagon  insiders,  it  tired  military  officers 
and  defense  specialists  speaJc  out 


Edited  bv  Winslow  T.  Wheeler 
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Mis-Measuring 
the  “Defense”  Budget 


What  Is  the  “Defense”  Budget? 


Category 

FY  2009 
(Request) 

DOD  (“Base”) 

518 

DOD  War  Funding 

70  +  87 

“Atomic  Energy  Defense  Activities”  (DOE) 

17 

“Defense  Related  Activities”  (GSA,  etc.) 

3 

Total  “National  Defense” 

698 

Homeland  Security  (DHS) 

40 

Veterans  Affairs  (DVA) 

91 

International  Affairs 

38 

Non-DOD  Military  Retirement 

12 

21%  of  Interest  on  the  Debt 

54 

Grand  Total 

934 
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DOD  Budget  1 947-2008 

(SBillions,  2008$) 

Source:  National  Defense  Budget  Estimates  for  FY  2009,  Office  of  the  Under  Secretary  of  Defense  (Comptroller)  March  2008 
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—  DoD  Budget  ($BN)  —  Cold  War  Average 
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Mis-Measurement  a  la  Conservatives 


%  of  GDP  for  DOD 
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Mis-Measurement  a  la  Liberals 


Defense  Spending  v.  Others 


800  -| 
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Mis-Measurement  a  la  Liberals 


Defense  Spending  v.  Others 
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All 

Appropriations 
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United  States  vs.  the  World 

($  Billions) 


MSS  2008  SIPRI 2008  CIA  2007  2008  National 

Defense 


e 


US  vs.  China,  Russia,  Cuba,  Iran,  North  Korea 

($  Billions,  2009  Dollars) 


□  Total  Defense  Spending  per  MSS  2008 


More  Money  Buys  a  Smaller  Force 
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Army  Division  Equivalents  and  Budget 

($Billions,  FY  2009  Dollars) 


Sources:  National  Defense  Budget  Estimates  for  FY  2009,  Office  of  the  Under  Secretary  of  Defense  (Comptroller)  March  2008,  Center  of  Military  History,  Historical  Perspective 

on  Force  Structure  Reductions  1946-1988,  Washington,  D.C.  1989,  Department  of  Defense  Annual  Reports,  and  Department  of  Defense  Appropriations  Bill  Reports  from  the 

House  Committee  on  Appropriations. 


<-Army  Division  Equivalents 


^Army  Budget  ($Billions) 


jfe  Jrp  'ft 
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Navy  Combat  Ships  and  Budget 

($Billions,  FY  2009  Dollars) 

Sources:  National  Defense  Budget  Estimates  for  FY  2009,  Office  of  the  Under  Secretary  of  Defense  (Comptroller)  March  2008 


d>  d>  d>  d>  d> 
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Active  Navy  Combat  Ships 


Navy  Budget 
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Air  Force  Tactical  Wings  and  Budget 

(Billions,  FY  2009  Dollars) 

Sources:  National  Defense  Budget  Estimates  for  FY  2009,  Office  of  the  Under  Secretary  of  Defense  (Comptroller)  March  2008  and  Air  Force 
Historical  Research  Agency  (Maxwell  AFB)  analysis,  “Force  Structure,  United  States  Army  Air  Forces  and  United  States  Air  Force.” 


/////// 


Active  Air  Force  Tactical  Wing 
Equivalents 


Air  Force  Budget 
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DOD  Budget  &  Active  Duty  Personnel 

(1947-2008) 

($Billions,  2008  Dollars) 


- DoD  Budget  ($BN)  - Active  Duty  Personnel  (xl,000) 

- Linear  (DoD  Budget  ($BN))  - Linear  (Active  Duty  Personnel  (xl,000)) 

More  Money  Buys  an  Older  Force 
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Figure  3-5  Updated 


Age  aadi  Inventory  of  Army  Ground  Combat  Vehicles 
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Figure  3-11  Updated 


Age  and  Inventory  of  Battle  Force  Skip;: 


Humber  of  Ships 
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Figure  3-21  Updated 


Age  and  Inventory  of  Air  Force  Fighter  and  Attack  Aircraft 


Actual  ■  FYDP  COO  Projection 
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And,  Less  Ready 
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Readiness 


Army  Tank  Miles:  Down  from  1 ,000  to  459 
Are  units  going  to  Iraq  “ready?” 

Naval  aviators  getting  17  hours/month. 


Air  Force  is  worse. 


More  Capability? 


F-22  Example 

-  Puny  Force 

-  Fewer  Resources  for  Pilot  Skill 

-  Aerodynamic  Disappointment 

-  How  much  “Stealth”  &  “Supercruise”? 

-  Unproven  BVR  Hypothesis 


Performance  in  Iraq  &  Afghanistan 


Two  Questions 
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Where  Did  the  $772  Billion  Go? 


•  $60+  billion  for  pork. 

•  $295  billion  for  hardware  overruns. 
-More  for  irrelevant  systems. 

•  Now  paying  $28  billion  more  for  4,000 
fewer  people. 
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More  Money  Won’t  Solve  It;  What  Will? 


•  Non-phony  reform  of  Pork  rules. 

•  Budget  Pause: 

-  Independent  assessment  of  all  weapons  &  personnel 
programs. 

-  Clean  the  financial  management  stables. 

•  Discriminate  in  favor  of  modern  veterans. 
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UNITED  STATES  ARMY  EVALUATION  CENTER 


Mission-Based  T&L 

Tutorial,  2  March  2009 


25th  Annual  NDIA 
T&E  Conference 


Chris  Wilcox 

US  Army  Evaluation  Center 
410-306-0475 

chris.wilcox1@us.army.mil 


Purpose 

•  To  present  and  overview  the  MBT&E 
methodology  (framework  and  process). 

•  To  engage  in  question/answer  discussions 
on  the  MBT&E  methodology. 

•  To  obtain  audience  feedback  on  the 
MBT&E  methodology. 
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Agenda 


1330:  Mission-Based  T&E  Background 
1340:  MBT&E  Framework 
1400:  Case  Study 
1415:  Procedure  -  Steps  1-5 

1500-1515:  Afternoon  Break 

1515:  Procedure  -  Steps  6-8 
1545:  Procedure  -  Steps  9-15 
1615:  Procedure  -  Steps  16-19 
1 630:  Discussions/Questions/Answers 
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MBT&E  Background 


Background  -  Why  MBT&E? 

•  Because  we  were  asked  to... 

-  DA/OSD-level  guidance: 

•  Address  recent  policy  initiatives,  such  as:  Section  231  Report; 
DOT&E/OUSD(AT&L)  T&E  Policy  Revisions  memo;  etc. 

-  “Show  impact  of  materiel  system  strengths/weaknesses  on  the 
operational  capabilities.” 

-  “Integrate  DT  and  OT  and  make  use  of  all  available  data.” 

•  Address  goals,  strategies  and  initiatives  in  DUSA-TEO  Strategic 
Plan,  2007. 

-  “Continuously  improve  T&E  policy  and  procedures.” 

-  “Increase  operational  realism  in  developmental  tests  to  improve  the 
likelihood  of  successful  operational  tests.” 

•  New  TEMP  format  and  DoD  5000  changes. 

-  “Integrated  T&E”  chapter  vs.  DT  and  OT  chapters. 
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Background  -  Why  MBT&E? 

•  Because  we  want  to  improve: 

-  The  way  we  do  our  job. . . 

•  Enable  robust  T&E  strategy  development  for  Joint 
networked  system-of-systems. 

-  The  way  we  support  the  warfighter. . . 

•  Answer  the  “so  what”  question.  (Complete  feedback 
loop  to  Capability-Based  Analysis.) 

•  Develop  way  to  link  system  performance  to  unit  and 
higher  unit  task  capabilities. 

-  The  way  we  support  the  materiel  developer. . . 

•  Scope  T&E  effort  earlier  in  the  acquisition  cycle. 


How  is  MBT&E  Implemented? 


Preliminary  Desigh  Feb  08:  1st  MBT&E  SummitBeg  >ilot  Projects  (3) 


80%  Design 


May  08:  Community  review  Ad<  lot  Projects  (10) 


Final  Design  Aug  08:  2nd  MBT&E  Summit  <j6nc|oing  Pilots  (15^ 


Published  ProcectarBec  08:  Procedure  Review 


Jan  09: 


New  T&E  Concepts  Using  MBT&E 


Lessons  Learned: 

MBT&E  framework  providing  context  of  operational  capability. 
MBT&E  process  is  executable  with  current  personnel  skill  set. 
Efficiencies  can  be  increased  through: 

•  Improved  tools  (templates,  IT,  training,  etc.);  and 

•  Combat  and  materiel  developer  participation. 


Background  -  MBT&E  Overview 

Mission-Based  Test  and  Evaluation 

is  a  methodology  that  focuses  T&E  on  the  mission 
task  capabilities  provided  to  the  warfighter.  It 
provides  a  framework  and  procedure  to: 

-  link  capabilities  to  the  attributes  of  the  materiel  system- 
of-systems; 

-  develop  evaluation  measures  that  assess  capabilities  and 
attributes; 

-and  link  the  evaluation  measures  to  all  available  data 
sources. 
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Agenda 


1330:  Introduction  and  Mission-Based  T&E  Background 

1340:  MBT&E  Framework 

1400:  Case  Study 

1415:  Procedure  -  Steps  1-5 

1500-1515:  Afternoon  Break 

1515:  Procedure  -  Steps  6-8 
1545:  Procedure  -  Steps  9-15 
1615:  Procedure  -  Steps  16-19 
1 630:  Discussions/Questions/Answers 


9 


MBT&E  Framework 
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Framework  Building  Block 

Capability1  -  The  ability  to  achieve  a  desired  effect 
[or  result,  outcome,  or  consequence  of  a  task2]  ... 


-  under  specified  standards  and  conditions 

-  through  a  combination  of  means  and  ways 

-  to  perform  a  set  of  tasks. 


1. CJCSI  31 70.01  F,  May  2007 

2.  Taken  from  JP  1-02,  Mar  2007,  definition  of  effect. 
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Framework  -  Task  Hierarchy 


MBT&E  Framework  —  v2 


Process/Products 


Commander’s  Task 
to  Subordinates 


- 1 - 

Transition  to  Allocating  Mission  Means 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


Commander’s  Task 
to  Subordinates 


- 1 - 

Transition  to  Allocating  SoS  Means 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


System  Attributes 


Transition  to  Allocating  Materiel  Means 

± 


Systems  Engineering 

•  Functional  Baseline 

•  Allocated  Baseline 

•  Product  Baseline 


f  AMSAA 

Excellence  in  Analysis 


Capability  =  Set  of  Tasks  +  Desired  Result 


Desired 
End  State 


Enables 


Operations  (Mission  Tasks) 

^ 

•  UJTLs 

Desired  Mission 

•  Service  TLs 

Task  Results 

•  Implied  Tasks 

Mission  Task  Capability 


System-of-Systems  Tasks 
Service  TLs 
Implied  Tasks 
Collective/Individual  Tasks 


T 


Desired  SoS 
Task  Results 


SoS  Task  Capability 


System  Performance 
•Functions  (shall  do) 
•  “shall  be’s” 

System  Performance 


Desired  System 
Performance  Results 


>- 


High  Level 
Tasks/Results 


Tasks/Results 
Specific  to 
System 


System 

Functions 
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Capability  and  Performance  Linked  to  Integrated  T&E 
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Agenda 


1330:  Introduction  and  Mission-Based  T&E  Background 
1340:  MBT&E  Framework 

1400:  Case  Study 

1415:  Procedure  -  Steps  1-5 

1500-1515:  Afternoon  Break 

1515:  Procedure  -  Steps  6-8 
1545:  Procedure  -  Steps  9-15 
1615:  Procedure  -  Steps  16-19 
1 630:  Discussions/Questions/Answers 
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Agenda 


1330:  Introduction  and  Mission-Based  T&E  Background 
1340:  MBT&E  Framework 
1400:  Case  Study 

1415:  Procedure  -  Steps  1-5 

1500-1515:  Afternoon  Break 

1515:  Procedure  -  Steps  6-8 
1545:  Procedure  -  Steps  9-15 
1615:  Procedure  -  Steps  16-19 
1 630:  Discussions/Questions/Answers 
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MBT&E  Procedure 
Steps  1-5 

Understand  the  Mission 
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Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  collect  information. 


understand  •  4  steps  to  understand  the  military  operations,  tasks,  task 
the  mission  capabilities  and  mission  context. _ 


0 


understand  •  2  steps  to  understand  the  components  and  attributes  of 
the  system  the  materiel  system-of-systems. _ 


•  1  additional  step  to  understand  the  mission  and  system 
linkages. 


o 


o 


design  the  •  7  steps  to  design  the  T&E  given  the  mission  and  system 
t&e _ understanding. _ 


determine  •  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
THE  RESULTS  data. 


REPORT 
THE  RESULTS 


1  step  to  format  and  report  the  results. 
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Procedure  - 

Collect  Information  (Step  1) 

Purpose:  Collect  available  information  to  gain  understanding  of: 

-  initiating  capability  gaps, 

-  mission  context  and  operational  conditions, 

-  mission  tasks  and  capabilities, 

-  system-of-systems  design  and  materiel  system  required  attributes. 

What  do  I  do? 

•  Work  through  capabilities  developer  and  materiel  developer  to 
obtain  the  available  information. 

-  Functional  Area,  Needs  &  Solution  Analyses,  Analysis  of  Alternatives, 
Requirements  Documents  (DoDAF  Views),  Threat  Assessments, 
Acquisition  Strategies,  Performance  Specifications,  etc. 

When  am  I  done? 

-  Checklist  of  documents  available/not  available  is  completed. 

-  Available  documents  Archived. 

-  Actions  to  obtain  copies  of  documents  available,  but  not  archived,  are 
presented  to  appropriate  IPTs. 
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Procedure  -  step  i 

Collect  Information  (Step  1)  side 


Currently 

building/maintaining 
document  list. 


Document  Title 

Functional  Area  Analysis 
Functional  Need  Analysis 
Functional  Solution  Analysis 
Analysis  of  Alternatives 
Operational  Mode  Summary/Mission  Profile 
Initial  Capabilities  Document 
DoDAF  OV-1  (Appendix  A) 

Capabilities  Documents 

DoDAF  OV-1  (Appendix  A) 

DoDAF  OV-2  (Appendix  A) 

DoDAF  OV-4  (Appendix  A) 

DoDAF  OV-5  (Appendix  A) 

DoDAF  OV-6C  (Appendix  A) 

DoDAF  SV-2  (Appendix  A) 

DoDAF  SV-4  (Appendix  A) 

DoDAF  SV-5  (Appendix  A) 

DoDAF  SV-6  (Appendix  A) 

COI/Cs 

Army  Universal  Task  List,  FM  7-15 
Unit  Mission  Training  Plan 
Support  Unit  Mission  Training  Plans 
Universal  Joint  Task  List,  CJCSM  3500 
System  Threat  Assessment  Report 
Technology  Development  Strategy 
Acquisition  Strategy 

Work  Breakdown  Structure 
Materiel  Fielding  Plan 
System  Performance  Specification 


Who  to  ask? 
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Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  understand  the  program  context. 


understand  the  •  4  steps  to  understand  the  military  operations,  tasks,  task 
mission_ capabilities  and  mission  context. 


understand  the  •  2  steps  to  understand  the  components  and  attributes  of 
system  the  materiel  system-of-systems. 


•  1  additional  step  to  understand  the  mission  and  system 
linkages. 


design  the  test  •  7  steps  to  design  the  T&E  given  the  mission  and  system 
AND  evaluation  understanding. _ 


DETERMINE  THE 
RESULTS 

•  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
data. 

REPORT  THE 
RESULTS 

•  1  step  to  format  and  report  the  results. 
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Procedure  - 

Define  the  Mission  Context  (Step  2) 

Purpose:  Define  the  overall  mission  area  context  that  the 

proposed  materiel  solution  is  being  developed  to  support. 

What  do  I  do? 

•  Determine  Operations/Mission/Tasks 

-  Develop  a  description  of  high-level  operations/mission/tasks  and 
their  desired  end  states/results, 

-  Determine  Joint,  network  and  SoS  construct,  and 

-  Determine  organizational  and  support  unit  construct. 

•  Determine  Operational  Conditions 

-  Determine  the  essential  elements  of  mission,  enemy,  terrain  and 
weather,  troops  and  support  available,  time  available,  and  civil 
considerations  (METT-TC). 

When  am  I  done? 

-  High-level  operations/missions/tasks  with  their  desired  end 
states/results  are  documented. 

-  Operational  conditions  (METT-TC  factors)  are  documented. 


Mission  Context  - 

Relation  to  Framework 

•  High-level  operations/mission/tasks. 

•  Task  desired  end  states/results. 
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Procedure 

Develop  the  Mission  Tasks  (Step  3) 

Purpose:  Develop  the  required  SoS  mission  tasks  and  link 
these  tasks  to  authoritative  tasks  lists. 

What  do  I  do? 

•  Document/Conduct  Mission  Analysis 

-  Develop  SoS  mission  task  threads  and  alternate  task  threads 
where  applicable.  (MS  project  or  similar  tool  can  be  used.) 

-  Determine  task  desired  end  states/results 

•  Link  to  Authoritative  Task  Lists 

-  Develop  linkages  between  the  tasks  identified  above  and  the 
appropriate  authoritative  task  lists.  (UJTL,  AUTL,  unit  Mission 
Training  Plans,  etc.) 

When  am  I  done? 

-  SoS  mission  tasks  with  their  desired  end  states/results  are 
documented. 

-  Links  to  associated  authoritative  tasks  are  documented. 
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Mission  Tasks  - 

Relation  to  Framework 

tasks  and  task  threads. 

Task  desired  end  states/results. 


MBT&E  Framework  -  v2 


JSAA 

Excellence  in  Analysis 


Process/Products 


Capability  = 


Set  of  Tasks  +  Desired  Result 


Commander’s  Task 
to  Subordinates 


- 1 - 

Transition  to  Allocating  Mission  Means 


Desired 
End  State 


Mission  Analysis 

•  Higher  Commander’s  Intent 

•  Restated  Mission 

•  Task  to  Subordinates 


Commander’s  Task 
to  Subordinates 


Operations  (Mission  Tasks) 

^ 

•  UJTLs 

Desired  Mission 

•  Service  TLs 

Task  Results 

•  Implied  Tasks 
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Procedure 

Develop  Supporting  Tasks  (Step  4) 


Purpose:  Develop  the  required  supporting  tasks  that  enable  the  execution 
of  the  SoS  mission  tasks. 


What  do  I  do? 

•  Determine  Conditional  Tasks  and  their  desired  effects/results. 

-  Conditional  tasks  are  performed  during  a  normal  mission  but  are  only 

required  due  to  some  influencing  condition. 

-  Examples:  avoid  threat  missile ,  extinguish  engine  fire,  reset  network  node, 
etc. 

•  Determine  Enabling  Tasks  and  their  desired  effects/results. 

-  Mission  enabling  tasks  are  conducted  in  order  to  enable  the  SoS  mission 
tasks  (task  developed  in  step  (3))  to  be  performed. 

-  Examples:  train,  deploy,  maintain,  etc. 

When  am  I  done? 

-  Conditional  tasks,  enabling  tasks  and  their  desired  end  states/results  are 
documented. 

-  Links  to  associated  authoritative  tasks  are  documented. 
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Procedure 

Identify  Task  Capabilities  (Step  5) 

Purpose:  Identify  and  associate  the  capabilities  required  to 
execute  the  SoS,  conditional  and  enabling  tasks. 

What  do  I  do? 

•  Identify  Required  Capabilities 

-  Identify  the  capabilities  required  to  support  each  task  with  a 
reference  to  applicable  requirements  documents.  (CDD,  CPD,  etc.) 

•  Associate  Tasks  with  Capabilities 

-  Link  the  capabilities  determined  above  with  the  mission,  conditional 
and  enabling  tasks  determined  in  steps  (3)  and  (4) 

When  am  I  done? 

-  Links  between  the  (SoS,  conditional  and  enabling  tasks)  and  the 
requirements  are  documented. 
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MBT&E  Procedure 

Steps  6-8 

Understand  the  System 
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Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  understand  the  program  context. 


understand  the  •  4  steps  to  understand  the  military  operations,  tasks,  task 
mission _ capabilities  and  mission  context. _ 


understand  the  •  2  steps  to  understand  the  components  and  attributes  of 
system_ the  materiel  system-of-systems. 


•  1  additional  step  to  understand  the  mission  and  system 
linkages. 


design  the  test  •  7  steps  to  design  the  T&E  given  the  mission  and  system 
and  evaluation  understanding. _ 


DETERMINE  THE 
RESULTS 

•  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
data. 

REPORT  THE 
RESULTS 

•  1  step  to  format  and  report  the  results. 
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Procedure 

Determine  SoS  Components  (Step  6) 


Purpose:  Identify  the  physical  components  of  the  materiel 
system  that  support  the  mission  tasks. 

What  do  I  do? 

•  Develop  a  materiel  system  description  starting  from  the  SoS 
level  and  breaking  down  into  components. 

•  Components  at  the  lowest  level  should  be  able  to  be  linked  to 
identifiable  functions  (shall  do’s)  and  enabling  attributes  (shall  be’s). 

•  Identify  the  functions  and  “shall  be’s”  of  the  materiel 
components. 

•  Option:  Develop  technology  risk  areas  for  pre-MS  B 
systems. 

When  am  I  done? 

-SoS  components  with  their  functions/shall  be’s  are  documented. 
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SoS  Components  - 

Relation  to  Framework 

*  SoS  Components 

*  SoS  Component  Functions  &  “Shall  be’s” 


Materiel  System  Functions:  An  activity  or  action  the  materiel  system  performs  in  support 
of  a  capability  or  part  of  a  capability. 

Materiel  System  “shall  be”:  An  attribute  the  materiel  system  possesses  that  enables  it  to 
perform  a  function,  for  example,  reliability. 
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Procedure 

Develop  System  Attributes  (Step  7) 


Purpose:  Identify  the  materiel  system  attributes  and  associate  them  with  the 
system  components. 


What  do  I  do? 

•  Identify  Attributes  Required 

-  Identify  the  materiel  system’s  attributes  required  to  support  the  component 
functions/shall  be’s  with  reference  to  applicable  requirements  documents. 
(CDD,  CPD,  Performance  Specification,  etc.) 


•  Associate  Components  with  Attributes 

-  Link  the  attributes  determined  above  with  the  system  components 
developed  in  step  6. 


When  am  I  done? 

•  Links  between  the  SoS  components  and  their  required  attributes  are 
documented. 
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Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  understand  the  program  context. 


understand  the  •  4  steps  to  understand  the  military  operations,  tasks,  task 
mission _ capabilities  and  mission  context. _ 


understand  the  •  2  steps  to  understand  the  components  and  attributes  of 
system  the  materiel  system-of-systems. 


1 

•  1  additional  step  to  understand  the  mission  and  system 
y/  linkages. 

design  THE  test  •  7  steps  to  design  the  T&E  given  the  mission  and  system 
AND  evaluation  understanding. 

DETERMINE  THE 
RESULTS 

•  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
data. 

REPORT  THE 
RESULTS 

•  1  step  to  format  and  report  the  results. 
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Procedure 

Associate  Capabilities  with  Attributes 

(Step  8) 

Purpose:  Develop  the  linkages  between  the  task  capabilities 
identified  in  step  (5)  and  the  materiel  system  component 
attributes  identified  in  step  (7). 

What  do  I  do? 

•  Link  the  system  attributes  (functions/shall  be’s)  to  the  task  capabilities. 

•  Determine  how  the  system  components  support  the  task  capability. 

•  Determine  redundant  system  support  capability. 

•  Determine  Mission  Enabling  Attributes 

•  Mission  Enabling  Attributes  are  system  enabling  attributes  that  are  not 
specific  to  a  particular  task  capability  -  they  address  all  tasks. 

When  am  I  done? 

-  Links  between  the  system  components  and  their  supported  tasks  are 
documented. 

-  Enabling  attributes  of  the  SoS  materiel  components  are  documented. 
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Capabilities  to  Attributes  - 

Relation  to  Framework 


*  Link  SoS  mission  task  capabilities  to  SoS  component 
attributes. 
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Agenda 


1330:  Introduction  and  Mission-Based  T&E  Background 

1340:  MBT&E  Framework 

1400:  Case  Study 

1415:  Procedure  -  Steps  1-5 

1500-1515:  Afternoon  Break 


1515:  Procedure  -  Steps  6-8 

1545:  Procedure  -  Steps  9-15 
1615:  Procedure  -  Steps  16-19 
1 630:  Discussions/Questions/Answers 


MBT&E  Procedure 
Steps  9-15 
Design  the  T&E 
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Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  understand  the  program  context. 


understand  the  •  4  steps  to  understand  the  military  operations,  tasks,  task 
mission _ capabilities  and  mission  context. _ 


understand  the  •  2  steps  to  understand  the  components  and  attributes  of 
system  the  materiel  system-of-systems. 


•  1  additional  step  to  understand  the  mission  and  system 
linkages. 


design  the  test  •  7  steps  to  design  the  T&E  given  the  mission  and  system 
and  evaluation  understanding. 


DETERMINE  THE 
RESULTS 

•  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
data. 

REPORT  THE 
RESULTS 

•  1  step  to  format  and  report  the  results. 
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Procedure 

Unconstrained  Operational  Conditions 

(Step  9) 

Purpose:  Develop  the  unconstrained  operational 
conditions  that  must  be  addressed  through  test 
and  evaluation. 

What  do  I  do? 

•  Determine  the  operational  factors  and  conditions  that  T&E 
needs  to  address  given: 

•  the  task  capability  required  and 

•  the  system  function/shall  be. 

When  am  I  done? 

-  Operational  conditions  (for  the  intersection  between 
mission  task  and  system  components  are  documented. 
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Operational  Conditions  - 

Linking  with  the  Matrix 


MISSION 

SYSTEM 

(Overall  mission(s),  ex.  "ART  3.3  Employ  Fires  to  Influence  the  Will,  Destroy,  Neutralize,  or  Supporess  Enemy  Forces." 

_ 

SYSTEM 

FUNCTION  or  CHARACTERS 

CPD  SYSTEM  TECHNICAL 

PERFORMANCE 

REQUIREMENTS 

1  MISSION  EXECUTION  TASKS  (Effectiveness) 

ENABLING  TASKS  (Suitability) 

CONDITIONAL  TASKS  (Survivability) 

ENABLING 

[Effectiveness  mission  task(s),  ex.  "ART  3.3.1  Conduct 
Lethal  Fire  Support"] 

[Support  Mission  task(s),  ex.  "ART  6.0 
Combat  Service  Support"] 

[Survivability  miss 

ex.  ART  5.3  C 
Survivability  Op 

lion  task(s), 

onduct 

erationsl 

1  [First  enabling  characteristic,  ex.  "Reliability"]  [Capab 

|  [Second  enabling  characteristic,  ex.  "Availability"]  [Ci 

1  [Additional  enabling  characteristics  examples:  Logistic 

isi 

ill 

lit 

\l\ 

2  £  ■£  A 

i 

■[First  sub-task  under  first 

|g^)5  P^r^rm^ire^isstons^in"^"  1 

^Degraded  Mode  on  the  Ml  09A6"]  | 

■[Second  sub-task  under  first 

Ic002  Conduct  Emergency  Missions"]  1 

([Capabilities  under  sub-task,  ex.  "KSA  | 

£ 

^Movements"]  [Capabilities  under 

■[First  sub-task  under  second 

■effectiveness  sub-task,  ex.  ”06-4-  1 

IGOOO  Perform  a  Survivability  Move  1 

((Paladin  Only)"]  [Capabilities  under  | 

£ 

£ 

£ 

First  suitability  sub-task,  ex.  "ART 

sub-task,  ex.  "06-4-C008  Store  1 

Ammunition  (Howitzer  Section)"] 

[Capabilities  under  sub-task] 

£ 

£ 

■[First  survivability  sub-task,  ex. 

■"ART  5.3.1  Protect  the  Friendly 

■Force  in  an  AO"]  [Capabilities 

(under  sub-task]  _ 

■[First  sub-task  under  first  survivability  1 

■sub-task,  ex.  "ART  5.3.2  Conduct  1 

(Nuclear,  Biological  and  Chemical  1 

■Defense"]  [Capabilities  under  sub-  | 

£s 

'f 

£ 

X 

[First  system  end-item,  ex.  "Self  Propelled  Howitzer"] 

or  characteristic,  ex.  "Reliability"].] 

[System  end-item  technical 
requirement,  ex.  "KPP  7.  Howitzer 
Reliability"] 

This 

task... 

X 

[First  end-item  sub-system,  ex.  "Armor"] 

[First  end-item  sub-system 
function  #1 ,  ex.  "Protect 
Crew  and  Systems''] 

[First  end-item  sub-system 
technical  requirement,  ex. 
"KPP2.  Force  Protection"] 

X 

[First  end-item  sub-system 
function  #2,  ex.  "Protect 
Crew  and  Systems"] 

[Additional  technical 
requirements] 

,  A  [Second  end-item  sub-system,  ex.  "Cannon"] 

[Second  end-item  sub¬ 
system  function.] 

[Additional  technical 
requirements] 

^TshowiJjB^ 

Under  these 

i 

riscic  am 

M  % 

/  [First  sub-system  of  Second 

/  end-item  sub-system,  ex. 

/ 

[First  sub-system  of  second 
end-item  sub-system 
function,  ex.  "Control  Tube 

T raverse/T urret  movement"] 

[Additional  technical 
requirements] 

[Additional  technical 
requirements] 

[Additional  technical 
requirements] 

conditions. 

/  [Second  sub-system  of 

second  end-item  subsystem, 
ex.  "Tube  Temperature 
Sensor"] 

etc. 

[additional  technical 
requirements] 

[additional  technical 
requirements] 

is  supported  by  this 

/ 

Etc. 

etc. 

[additional  technical 
requirements] 

component/function . . . 

[Second  end-item,  ex.  "Resupply  Vehicle"] 

[additional  technical 
requirements] 

[ 

r 

etc. 

[Additional  technical 
requirements] 

[Additional  technical 
requirements] 

etc. 

etc. 

[Additional  technical 
requirements] 

etc. 

[Additional  technical  requirements] 

etc. 

[Additional  technical 
requirements] 

[Additional  technical 
requirements] 

L 

etc. 

etc. 

[Additional  technical 
requirements] 

tjzt 


Process 

Develop  the  Evaluation  Strategy 

(Step  10) 

Purpose:  Develop  a  summary  description  of  the 
evaluation  to  support  an  early  strategy 
coordination  and  review. 

What  do  I  do? 

•  Develop  the  early  strategy  review  brief  from 
the  mission,  task,  and  system  worksheets 
developed  in  steps  (2)  through  (9). 

When  am  I  done? 


-  Early  strategy  review  brief  is  prepared. 
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Process 

Develop  Evaluation  Measures 

(Step  11) 

Purpose:  Develop  the  evaluation  measures. 

What  do  I  do? 

•  Develop  measures  supporting  the  evaluation  of: 

•  task  capabilities  (task  capability  measures),  and 

•  system  attributes  (materiel  performance  measures). 

•  Complete  linkages  from  measure  -to-  system  -to-  task. 

•  Develop  linkages  between  measures  and  COIs/Criteria. 

When  am  I  done? 

-  Task  capability  and  materiel  performance  measures  are 
documented. 
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Evaluation  Measures  - 

Relation  to  Framework 

*  Task  Capability  Measures 

*  Materiel  Performance  Measures 
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Operational  Measures  side 

•  Describe  operational  T&E  measures  linked 
to  the  tasks. 

•  Describe  how  operational  conditions  are 
used  to  define  data  requirements 
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Technical  Measures 


SIDE 


•  Describe  technical  T&E  measures  linked 
to  the  materiel  system  attributes  (functions 
and  characteristics). 

•  Describe  how  operational  conditions  are 
used  to  define  data  requirements 
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Design  the  T&E 

Assign  Measures  to  Data  Sources 

(Step  12) 


Purpose:  Identify  the  sources  of  data  to  support  the  analysis  of 
the  evaluation  measures. 


What  do  I  do? 

•  Assign  one  or  more  data  sources  to  each  evaluation  measure. 

•  Review  data  source  matrix  to  determine: 

•  T&E  execution  risk  by  assessing  critical  data  sources; 

•  Developmental  risk  by  assessing  when  critical  technologies  are 
demonstrated;  and 

•  Determine  appropriate  use  of  M&S. 

When  am  I  done? 

-  Data  sources  supporting  each  measure  are  documented. 

-  T&E  effort  and  program  execution  risk  issues  are  identified  and 
coordinated  with  the  appropriate  IPTs. 
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Data 

Relation 


*  Data  Sources 


Sources  - 

to  Framework 


•  Linked  to  Evaluation  Measures 


46 


Link  to  Data  Sources 


SIDE 


•  Describe  the  process  used  to  link  the 
evaluation  measures  to  the  appropriate 
data  sources. 

•  Describe  how  to  do  first  cut  on  what 
operational  conditions  are  provided  by  the 
tests  (data  sources). 
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Design  the  T&E 

Constrained  Operational  Conditions 

(Step  13) 

Purpose:  Develop  the  constrained  operational  conditions  by 
looking  at  the  conditions  that  can  be  addressed  by  the 
identified  data  sources. 

What  do  I  do? 

•  Determine  the  operational  conditions  that  can  be 
addressed  by  the  identified  data  sources.  These  are  the 
“constrained”  operational  conditions. 

•  Determine  the  T&E  limitations  by  comparing  the 
constrained  vs.  unconstrained  operational  conditions  (step 
9). 

When  am  I  done? 

-  T&E  limitations  caused  by  the  lack  of  ability  to  address 
all  operational  conditions  documented. 

-  Mitigation  effort(s)  to  lesson  impact  of  T&E  limitations 
are  documented. 


Design  the  T&E 

Develop  Data  Source  Requirements 

(Step  14) 


Purpose:  Develop  data  elements  from  each  linked  data 
source. 

What  do  I  do? 

•  Develop  detailed  measure  design. 

•  Determine  data  elements  required  from  the  data  source. 

•  Determine  the  operational  conditions  required  for  each  run, 
sortie  or  sample. 

When  am  I  done? 

-  Data  source  requirements  documented  and  coordinated 
with  the  appropriate  executing  test  agent  (contractor, 
government  range,  independent  test  facility,  M&S,  etc.) 
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Constrained  Op  Conditions 

SIDE 


•  Describe  DOE 
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Design  the  T&E 

Develop  T&E  Databases  (Step  15) 

Purpose:  Develop  database  architecture  to  enable  efficient 
delivery,  formatting  and  analysis  of  delivered  data. 

What  do  I  do? 

•  Develop  an  evaluation  data  model  from  the  task 
description,  enabling  attribute,  measure  description 
worksheets. 

•  The  evaluation  data  model  is  a  representation  of  the  information 
and  data  assets  required  to  evaluate  the  system  expressed  in 
terms  of  entities  and  relationships  between  entities. 

•  Provide  evaluation  data  model  results  to  the  tester. 

•  The  evaluation  data  model  will  ensure  properly  documented  data 
for  communication  between  the  evaluator  and  the  tester.. 

When  am  I  done? 

-  T&E  database  design  is  documented. 
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MBT&E  Current  Toolbox 


MBT&E  Worksheets 


MBT&E  Matrix 


Used  to  build  test  and  evaluation  plan. 


Mission 

Summary 


Evaluation 

Measure 

Description 


-  . - 


ATEC 


Task  and 
Enabling 
Attribute 
Descriptions 


-  Maintains  status  and  tracks 
changes  during  execution. 
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Agenda 


1330:  Introduction  and  Mission-Based  T&E  Background 

1340:  MBT&E  Framework 

1400:  Case  Study 

1415:  Procedure  -  Steps  1-5 

1500-1515:  Afternoon  Break 


1515:  Procedure  -  Steps  6-8 
1545:  Procedure  -  Steps  9-15 

1615:  Procedure  -  Steps  16-19 
1 630:  Discussions/Questions/Answers 


MBT&E  Procedure 
Steps  16-19 

Determine  and  Report  the 

Results 
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Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  understand  the  program  context. 


understand  the  •  4  steps  to  understand  the  military  operations,  tasks,  task 
mission _ capabilities  and  mission  context. _ 


understand  the  •  2  steps  to  understand  the  components  and  attributes  of 
system  the  materiel  system-of-systems. 


•  1  additional  step  to  understand  the  mission  and  system 
linkages. 


design  the  test  •  7  steps  to  design  the  T&E  given  the  mission  and  system 
and  evaluation  understanding. _ 


DETERMINE  THE 
RESULTS 

•  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
data. 

REPORT  THE 
RESULTS 

•  1  step  to  format  and  report  the  results. 
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Execution 

(Steps  16  through  18) 

CONDUCT  TEST  AND  GATHER  DATA  (Step  16) 

Purpose:  Execute  the  planned  data  source  activities  and  gather  the  data  for  analysis. 
What  do  I  do? 

•  Execute  test,  run  M&S,  record  data. 

•  Review  data  for  integrity  and  authentication. 

•  Adjust  T&E  program  based  on  impacts  of  changes  in  schedule  and  system  design. 

PERFORM  DATA  ANALYSIS  (Step  17) 

What  do  I  do? 

•  Data  is  analyzed  according  to  the  procedures  identified  in  step  1 1  and  14. 

•  Performance  results  are  compared  to  standards  identified  in  steps  5  (task  capabilities) 
and  7  (system  attributes). 


GENERATE  EVALUATION  RESULTS  (Step  18) 

What  do  I  do? 

•  Determine  materiel  system  attribute  performance. 

•  Determine  SoS  task  capabilities  and  limitations. 

-  Determine  task  capability  C&L  directly  from  task  capability  measure  results. 

-  Determine  task  capability  C&L  based  on  system  attribute  measure  results. 

•  Determine  task  C&L  impact  on  high-level  mission  task  capabilities. 

-  Determine  ability  to  achieve  desired  end  state  directly  from  capability  measures. 

-  Determine  ability  to  achieve  desired  end  state  from  SoS  task  capability  C&Ls. 
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Procedure  -  Overview 


•19  steps  divided  into  5  major  purpose  areas. 

•  1  Pre-step  to  understand  the  program  context. 


understand  the  •  4  steps  to  understand  the  military  operations,  tasks,  task 
mission _ capabilities  and  mission  context. _ 


understand  the  •  2  steps  to  understand  the  components  and  attributes  of 
system  the  materiel  system-of-systems. 


•  1  additional  step  to  understand  the  mission  and  system 
linkages. 


design  the  test  •  7  steps  to  design  the  T&E  given  the  mission  and  system 
and  evaluation  understanding. _ 


DETERMINE  THE 
RESULTS 

•  3  steps  to  generate,  collect,  analyze,  and  evaluate  the 
data. 

REPORT  THE 
RESULTS 

•  1  step  to  format  and  report  the  results. 
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Reporting 

Generate  Evaluation  Report  (Step  19) 


Purpose:  To  generate  the  evaluation  report  which  will  document  the 
evaluation  conclusions. 

What  do  I  do? 

•  Generate  system  performance  and  SoS  task  C&L  conclusions. 

•  Generate  summary  of  key  C&Ls. 

•  Generate  effectiveness,  suitability  and  survivability  conclusions. 

When  am  I  done? 

•  System  performance,  (strengths  and  weaknesses),  and  the  impact  they 
had  on  the  task  capabilities  are  documented. 

•  SoS  task  C&Ls  and  high-leve  task  C&Ls  are  documented. 

•  Overall  summary  of  effectiveness,  suitability  and  survivability  is 
documented. 
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Mission  Element 


Task  Type  Definitions 


Air  Assault  | 

Resupply  a  Maneuver  Brigade 

Mission  execution  tasks. 


-  Tasks  that  describe  a  discrete 
action  that  the  unit  (system  and  its 
operators)  must  perform  in  order  to 
accomplish  its  main  mission. 


Conditional  mission  tasks. 

-  Tasks  that  are  performed  during 
the  mission  that  become  required 
due  to  some  influencing  condition. 

Mission  enabling  tasks. 

-  Tasks  that  enable  the  mission 
execution  and  conditional  tasks  to 
be  performed.  They  usually  occur 
before  or  after  the  mission. 


Conditional  Mission  Tasks 

Level  1  Tasks 

Level  2  Tasks 

1  Acknowledge  Missile  Launch*  I 

t 

Avoid  IR  Threat - i— < 

1  Maintain  SA  of  Threat  I 

Control  Aircraft  Fliqht  I 

1  Deploy  Countermeasures  1 

Mission  Enabling  Tasks 


Leve 11  Tasks  Level  2  Tasks 


< 


Enabling  Attributes 

-  System  attributes  that  affect  all 
tasks. 


Normally 

aggregated 

into 

Effectiveness 

and 

Survivability 


Normally 

aggregated 

into 

Suitability 
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Evaluation  Report 

SIDE 

•  Describe  reporting  of  overall  capabilities 
and  limitations 

•  Describe  roll-up  into  effectiveness, 
suitability  and  survivability 
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Agenda 


1330:  Introduction  and  Mission-Based  T&E  Background 

1340:  MBT&E  Framework 

1400:  Case  Study 

1415:  Procedure  -  Steps  1-5 

1500-1515:  Afternoon  Break 

1515:  Procedure  -  Steps  6-8 
1545:  Procedure  -  Steps  9-15 
1615:  Procedure  -  Steps  16-19 

1 630:  Discussions/Questions/Answers 
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Discussions 

Questions 

Answers 
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MBT&E  Point  of  Contact 


Christopher  Wilcox 

US  Army  Evaluation  Center 
ATTN:  TEAE-AVA  (Mr.  Chris  Wilcox) 
4120  Susquehanna  Ave. 
Aberdeen  Proving  Ground,  MD  21005 

Office:  (410)  306-0475 
Mobile:  (410)  652-0168 
Fax:  (410)306-0398 

chris.wilcoxl  @us.army.mil 
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Mission-Based 
Test  &  Evaluation 


NDIA  25th  Annual 
T&E  Conference 


Mr.  Christopher  Wilcox 
3  March  2009 


\\ 


Purpose 


•  Inform  T&E  community  peers  of  the 

effort  to  design  and  implement  a  mission- 
based  test  and  evaluation  (MBT&E) 
methodology. 


Solicit  questions  and  comments  from 
peers  to  help  improve  the  MBT&E 
methodology. 


2 


Agenda 

•  Why  was  MBT&E  developed? 

•  What  is  MBT&E? 


•  What  does  MBT&E  do? 


•  How  is  MBT&E  implemented? 

•  Where  is  MBT&E  headed? 
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Why  was  MBT&E  developed? 


•  Networked  system-of-systems 

•  Address  Acquisition  Initiatives 

•  Provide  “feedback”  to  capabilities 
integration  and  development 

“We  will  continue  to  examine  and  challenge  our  most  basic 
institutional  assumptions,  organizational  structure  paradigms,  policies, 
and  procedures  to  better  serve  the  Army.  ” 

CG,  ATEC  Commander’s  Priorities  for  FY  10-15 
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What  is  MBT&E? 


Mission-Based  Test  and  Evaluation 

is  a  methodology  that  focuses  T&E  on  the 
capabilities  provided  to  the  warfighter.  It  provides  a 

framework  and  procedure  to: 

-  link  materiel  system  attributes  to  the  operational 
capabilities; 

-  examine  the  SoS  required  to  enable  the  operational 
capability;  and 

-  enable  synergistic  use  of  all  available  data  sources. 
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Framework  Building  Block 

Capability1  -  The  ability  to  achieve  a  desired  effect 
[or  result,  outcome,  or  consequence  of  a  task2]  ... 

-  under  specified  standards  and  conditions 

-  through  a  combination  of  means  and  ways 

-  to  perform  a  set  of  tasks. 


1. CJCSI  31 70.01  F,  May  2007 

2.  Taken  from  JP  1-02,  Mar  2007,  definition  of  effect. 
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MBT&E  Framework 
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Task  Capability  Linked  to  System  Performance 

<  T&E  Planning 

T&E  Execution  > 


MBT&E  Framework  Example 


Capability  and  Performance  Linked  to  Integrated  T&E 
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MBT&E  Process 


•  Process  divided  into  steps. 

•  Steps  divided  into  5  major  purpose  areas. 


UNDERSTAND  THE 
MISSION 

•  Mission  and  task  context. 

UNDERSTAND  THE 
SYSTEM 

•  Materiel  components  and  attributes. 

*  Linkages  between  mission  and  materiel. 

DESIGN  THE  TEST 
AND  EVALUATION 

•  Test  design  and  evaluation  measures. 

DETERMINE  THE 
RESULTS 

•  Execute  test  and  evaluation. 

REPORT  THE 
RESULTS 

•  Format  and  report  the  results. 
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What  does  MBT&E  do? 


•  Operational  capabilities  and  limitations. 

•  Materiel  system  performance  and  effect  on 
operational  capabilities. 

•  Effectiveness,  suitability  and  survivability 
based  on  task. 
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Report  Example 
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How  is  MBT&E  Implemented? 


Preliminary  Design  Feb  08:  1st  MBT&E  Summit  <C  [egin  Pilot  Projects  (3) 


80%  Design 


May  08:  Community  review  <C  d  Pilot  Projects  (10) 


Final  Design 


Aug  08:  2nd  MBT&E  Summit 


Ongoing  Pilots  (15) 


Published  Procedure^  Dec  08:  Procedure  Review 


Jan  09: 


New  T&E  Concepts  Using  MBT&E 


Lessons  Learned: 

MBT&E  framework  providing  context  of  operational  capability. 
MBT&E  process  is  executable  with  current  personnel  skill  set. 
Efficiencies  can  be  increased  through: 

•  Improved  tools  (templates,  IT,  training,  etc.);  and 

•  Combat  and  materiel  developer  participation. 
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Where  is  MBT&E  headed? 


Synchronize  with 

capabilities-based 

analysis. 

Synchronize  with 
systems  engineering. 

Collaborative 

environment. 
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Summary 

•  MBT&E  methodology  developed. 

•  Positive  results  and  path  forward 
toward  increased  efficiencies. 

•  Aligning  the  efforts  of  the  capabilities 
developer,  materiel  developer,  and 
independent  T&E. 
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Christopher  Wilcox 


US  Army  Test  and  Evaluation  Command 
US  Army  Evaluation  Center 
ATTN:  TEAE-MAA  (Mr.  Chris  Wilcox) 
4120  Susquehanna  Ave. 
Aberdeen  Proving  Ground,  MD  21005 


Office:  (410)  306-0475 
Fax:  (410)  306-0398 
chris.wilcoxl  @us. army.mil 


25th  Annual  National 

Test  &  Evaluation  Conference 


The  Age  of  Alternative  Energy 

Keith  Williams 

President  and  CEO 
Underwriters  Laboratories  Inc. 


March  3,  2009 
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Walking  the  Line  with  Title  10: 
Implementation  Strategies  for 
Integrated  Testing 


Beth  Wilson 

Industry  Co-Chair  NDIA  System  Engineering  Division,  DT&E  Committee 
Test  Architect,  Raytheon  Company 

Darlene  Mosser-Kerner 

Government  Chair  NDIA  System  Engineering  Division,  DT&E  Committee 
Developmental  Test  &  Evaluation 
OUSD(AT&L)/Systems  &  Software  Engineering 
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Why  Integrated  Testing? 


500002 

a.  The  fundamental  purpose  of  T&E  is  to  provide  knowledge  to  assist  in  managing  the  risks 
involved  in  developing,  producing,  operating,  and  sustaining  systems  and  capabilities.  T&E 
measures  progress  in  both  system  and  capability  development.  T&E  provides  knowledge  of 
system  capabilities  and  limitations  to  the  acquisition  community  for  use  in  improving  the  system 
performance,  and  the  user  community  for  optimizing  system  use  in  operations.  T&E  expertise 
must  be  brought  to  bear  at  the  beginning  of  the  system  life  cycle  to  provide  earlier  learning  about 
the  strengths  and  weaknesses  of  the  system  under  development.  The  goal  is  early  identification 
of  technical,  operational,  and  system  deficiencies,  so  that  appropriate  and  timely  corrective 
actions  can  be  developed  prior  to  fielding  the  system. 


[  strongly  believe  that  CT&E  should  be  a  process  of  con¬ 
firmation  and  not  one  of  discovery.  Unfortunately,  OT&E 
is  too  often  the  place  where  performance  shortcomings 
Dr.  Charles  E.  McQueary  ami  new  failure  modes  are  discovered.  When  problems 
Director  of  Operational  Test  and  Evaluation  are  discovered  late  In  the  acquisition  process,  the  cost  to 

Defense  AT&L:  January-February  2008  f[x  these  Prob[ems  is  much  than  if  they  were  di^ 

covered  earlier.  In  addition,  the  time  lost  when  problems 
are  found  at  this  stage  can  be  substantial— and  when  our 
forces  need  a  new'  capability,  the  latter  penalty  may  be 
even  more  substantial  than  increased  cost. 

Navy  OT&E  Framework  Integrated  Test  Methodology 

Robust  testing  minimizes  “surprises”  when  the  product  is  sent  to  the  war  fighter  and  ensures  the 
specified  capabilities  are  evaluated  in  the  operational  environment.  Risk  is  reduced  by  bringing  all 
testing  agents  together  early  in  the  process  to  ensure  capabilities  are  tied  to  mission,  mission 
oriented  testing  is  conducted,  system  anomalies/deficiencies  are  identified  early  in  the  process,  and 
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of  technical,  operational,  and  system  deficiencies,  so  that  appropriate  and  timely  corrective 
actions  can  be  developed  prior  to  fielding  the  system.  ___  _ 

OT&E  should  be  a  process  of  confirmation 
and  not  one  of  discovery 

Dr.  Charles  E.  McQueary  and  new  failure  modes  are  discovered.  When  problems 
Director  of  Operational  Test  and  Evaluation  are  discovered  late  in  the  acquisition  process,  the  cost  to 
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covered  earlier.  In  addition,  the  time  lost  when  problems 

are  found  at  this  stage  can  be  substantial— and  when  our 
forces  need  a  new'  capability,  the  latter  penalty  may  be 
even  more  substantial  than  increased  cost. 

Navy  OT&E  Framework  Integrated  Test  Methodology 

Robust  testing  minimizes  “surprises'5  when  the  product  is  sent  to  the  war  fighter  and  ensures  the 
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Why  Integrated  Testing? 


DSB  Task  Force  on  DT&E  May  2008  Report 

Findings 

The  changes  in  the  last  15  years,  when  aggregated,  have  had  a  significant  negative  impact  on 
DoD's  ability  to  successfully  execute  increasingly  complex  acquisition  programs.  Major 
contributors  include  massive  workforce  reductions  in  acquisition  and  test  personnel,  a  lack  of  up- 
to-date  process  guidance  in  some  acquisition  organizations,  acquisition  process  changes,  as  well 
as  the  high  retirement  rate  of  the  most  experienced  technical  and  managerial  personnel  in 
government  and  industry  without  an  adequate  replacement  pipeline, 

*  Major  personnel  reductions  have  strained  the  pool  of  experienced  government  test 
personnel 

The  attacks  of  September  11,  2001,  ushered  in  a  new  era  of  warfighting  with  the  Global  War  on 
Terrorism.  Significant  priority  was  given  to  finding  more  efficient  ways  to  deliver  new 
capabilities  to  the  Combatant  Commanders  for  use  against  quickly  adapting  threats  Rigorous 
T&E  before  deployment  was  sometimes  sacrificed  to  meet  schedule  demands. 

Navy  OT&E  Framework  Integrated  Test  Methodology 

all  data  are  shared.  Cost  is  reduced  by  the  sharing  of  resources,  elimination  of  duplicative  testing, 
and  the  early  identification  and  correction  of  deficiencies.  Schedule  compression  is  achieved  by 
combined  vs.  sequential  testing  and  the  sharing  of  high- demand  testing  assets.  None  of  these 
objectives  can  be  achieved  without  the  cooperation  of  all  parties  and  commitment  to  a  “teauf1 
approach  between  the  program,  office.  OT.  DT.  and  contractor  personnel  involved. 


Need  integrated  testing  to  meet  cost/schedule  demands 
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Integrated  Test  Definitions:  DoD 


DAG  Chapter  9 

9.3.3.  Combined  DT&E  and  OT&E 


OSD  McQueary/Young  Memo  22  Dec  2007 

*  Developmental  and  operational  test  activities  shall  be  integrated  and  seamless 
throughout  the  system  life  cycle.  As  technology,  software,  and  threats  change, 
follow-on  T&E  should  be  used  to  assess  current  mission  performance  and  inform 
operational  users*  during  the  development  of  new  capability  requirements. 


Whenever  feasible.  DT&E  and  OT&E  events  should  be  combined,  if  that  supports  technical  and 
operational  test  objectives  to  gain  the  optimum  amount  of  testing  benefit  for  reasonable  cost 
and  time.  The  user  community  should  be  involved  early  in  test  planning  to  ensure  the  statement 
of  desired  capabilities  is  interpreted  correctly  and  tested  realistically.  Certain  events  can  be 
organized  to  provide  information  useful  to  developmental  and  operational  evaluators  and  lend 
themselves  to  the  combined  DT  and  OT  approach.  The  concept  is  to  conduct  a  single, 
combined  test  program  that  produces  credible  qualitative  and  quantitative  information  that  can 
be  used  to  address  developmental  and  operational  issues.  Examples  of  this  approach  include 
combined  DT  and  OT  events,  or  piggybacking  an  operational  assessment  onto  a  developmental 
test.  Likewise,  developmental  testing  data  requirements  can  be  accommodated  by  an 
operational  test.  This  approach  can  reduce  the  time  and  expense  of  conducting  dedicated  OT 
events  that  replicate  DT  events,  or  vice  versa,  yet  still  provide  adequate  technical  risk  reduction. 

The  developmental  and  operational  testers  can  develop  a  test  management  structure  to  share 
control  of  the  combined  events.  Combined  DT  and  OT  events  and  test  data  requirements  must 
be  identified  early  to  prevent  unnecessary  duplication  of  effort  and  to  control  costs.  It  is 
important  that  neither  the  DT&E  nor  OT&E  objectives  are  compromised  in  designing  combined 
events.  For  further  explanation  of  this  combined  strategy,  refer  to  the  DAU  Test  and  Evaluation 
Management  Guide. 

OSD  McQueary/Finley  Memo  25  Apr  2008 

Integrated  testing  is  the  collaborative  planning  and  collaborative  execution 
of  test  phases  and  events  to  provide  shared  data  in  support  of  independent 
analysis,  evaluation  and  reporting  by  all  stakeholders  particularly  the 
developmental  (both  contractor  and  government)  and  operational  test  and 
evaluation  communities. 
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Integrated  Test  Definitions:  Services 


Army  DA  PAM  73-1  *  30  May  2003 

Integrated  testing  and  evaluation 

A  T&E  strategy  that  reduces  the  multiple  and  redundant  products  and  processes,  and  encompasses  the  development  of 
a  single  integrated  system  evaluation  plan  and  a  single  integrated  test/simulation  strategy,  leading  to  a  single  system 
evaluation  report  for  the  customer.  The  process  also  increases  the  use  of  contractor  data  for  evaluation  and  expands  the 
use  of  M&S  with  the  goal  of  reducing  T&E  costs.  Integrated  T&E  strategies  may  include  combined  DT-'OT  events 
where  appropriate. 

Integrated  DTVOT 

Integrated  DT/OT,  a  special  case  of  a  Combined  DT/OT,  is  a  single  phased  event  that  generates  data  to  address 
developmental  and  operational  issues  simultaneously  under  operational  conditions.  The  execution  strategy  for  this  event 
is  based  on  the  requirements  of  the  program. 

Navy  OT&E  Framework  Integrated  Test  Methodology 

IT  is  a  cooperative  approach  to  T&E  where  CT.  DT.  and  OT  entities  work  to  blend 
or  integrate  the  T&E  requirements  throughout  the  defense  acquisition  process. 

Integration  of  CT.  DT.  and  OT  does  not  involve  the  analysis  and  repotting  aspects 
of  T&E,  which  remain  solely  under  the  purview  of  the  respective  CT,  DT,  or  OT 
organization. 
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Integrated  Testing — The  harmonization  of  all  types  of  tests  and  evaluations  that  are  planned  and 
integrated  as  early  as  possible  into  ail  efficient  continuum,  efficiently  phased  and  resourced  over  time,  and 
reported  collaboratively  in  order  to  achieve  greater  test  efficiency,  reduced  cost,  and  schedule  savings 
without  compromising  the  objectives  and  needs  of  the  participating  test  organizations, 

Integrated  Test  Team  (ITT) — A  cross -functional  team  of  empowered  representatives  from  multiple 
disciplines  and  organizations  and  co -chaired  by  operational  testers  and  the  program  manager.  The  ITT  is 
responsible  for  developing  the  T&E  strategy  and  TEMP,  assisting  the  acquisition  community  with  T&E 
matters,  and  guiding  the  development  of  test  plans  that  are  integrated.  Note:  The  ITT  is  the  Air  Force 
equivalent  to  the  T&E  Working  Integrated  Product  Team  (T&E  WIPT)  described  in  the  Defense 

Acquisition  Guidebook.  8 
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Integrated  Test  Definitions:  Services 


and  encompasses  the  development  of 


Army  DA  pam  7qL-1  - 

a T&Emate  Single  integrated  test  strategy 

a  single  integrated  system  evaluation  plan  and  a  single  integrated  test. 'simulation  strategy,  leading  to  a  single  system 
evaluation  report  for  the  customer.  The  process  also  increases  the  use  of  contractor  data  for  evaluation  and  expands  the 
use  of  M&S  with  the  goal  of  reducing  T&E  '  "  '  '  A  '  _ 

where  appropriate.  Generates  data  to  address  developmental 

Integrated  DIVOT 

Integrated  DT/OT,  a  special  case  of  a  Con 
developmental  and  operational  issues  sunultai 
is  based  on  the  requirements  of  the  program. 


and  operational  issues  simultaneously 
under  operational  conditions 


Navy  OT&E  Framework  Integrated  Test  Methodology 

IT  is  a  cooperative  approach  to  T&E  where  CT.  DT.  and  OT  er 
or  integrate  the  T&E  requirements  throughout  the  defense  acqu 
Integration  of  CT.  DT.  and  OT  does  not  involve  the  analysis  ai 
of  T&E,  which  remain  solely  under  the  purview  of  the  respect! 
organization. 


CT,  DT,  and  OT  entities  work  to  blend 
or  integrate  the  T&E  requirements 


Harmonization  of  all  types  of  tests 


Air  Force  AFI99-103  26  FEBRUARY  2008 
Integrated  Testing — The  han 
integrated  as  early  as  possible  i 
reported  collaboratively  in  order  to  achieve  greater  test  efficiency,  reduced  cost,  and  schedule  savings 
without  comprom  ,  ,  ,  ,  ,  ,  .  ,  ...  .  ,  , . 

Integrated  as  early  as  possible  into  an  efficient  continuum 

Integrated  Test  ^  .. 

disciplines  and  organizations  and  co -chaired  by  operational  testers  and  the  program  manager.  The  ITT  is 

responsible  tor  developing  the  t&i  Development  of  test  plans  that  are  integrated 

matters,  and  guiding  the  develop m  ~  1  ° 

equivalent  to  the  T&E  Working  Integrated  Product  Team  (T&E  WIPT)  described  in  the  Defense 
Acquisition  Guidebook. 
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What  Does  Title  10  Say? 


(d)  Impartiality  of  Contractor  Testing  Personnel. —  In  the  case  of  a  major  defense  acquisition 
program  (as  defined  in  subsection  (a)(2)),  no  person  employed  by  the  contractor  for  ihe  system  being 
tested  may  be  involved  in  the  conduct  of  the  operational  test  and  evaluation  required  under  subsection 
(a).  The  limitation  in  the  preceding  sentence  does  not  apply  to  the  extent  that  the  Secretary  of  Defense 
plans  for  persons  employed  by  that  contractor  to  be  involved  in  the  operation,  maintenance,  and  support 
of  the  system  being  tested  when  the  system  is  deployed  in  combat. 

(e)  Impartial  Contracted  Advisory  and  Assistance  Services. — 

(1)  The  Director  may  not  contract  with  any  person  for  advisory  and  assistance  sendees  with  regard 
to  the  test  and  evaluation  of  a  system  if  that  person  participated  in  (or  is  participating  in)  the 
development,  production,  or  testing  of  such  system  for  a  military  department  or  Defense  Agency 
(or  for  another  contractor  of  the  Department  of  Defense). 

(2)  The  Director  may  waive  the  limitation  under  paragraph  (1)  in  any  case  if  the  Director 
determines  in  writing  that  sufficient  steps  have  been  taken  to  ensure  the  impartiality  of  the 
contra|ctor  in  providing  the  services.  The  Inspector  General  of  the  Department  of  Defense  shall 
review  each  such  waiver  and  shall  include  in  the  Inspector  General’s  semi-annual  report  an 
assessment  of  those  waivers  made  since  the  last  such  report. 

^  (A)  A  contractor  that  has  participated  in  (or  is  participating  in)  the  development,  production, 
or  testing  of  a  system  for  a  military  department  or  Defense  Agency  (or  for  another  contractor  of 
the  Department  of  Defense)  may  not  be  involved  (in  any  way)  in  the  establishment  of  criteria 
for  data  collection,  performance  assessment,  or  evaluation  activities  for  the  operational  test 
and  evaluation. 


conduct  of 
OT&E 

establishing 
criteria  for  OT&E 

OT&E 

evaluation 


(B)  The  limitation  in  subparagraph  (A)  does  not  apply  to  i 
such  development,  production,  or  testing  solely  in  testing  j 

10  USC  23 


NB:  This  unofficial  compilation  of  the  U.S.  Code  is  current  as  of  Jan.  3 


Contractor  cannot  be  involved  in: 

•  OT&E  conduct 

•  Establishing  OT&E  criteria 

•  OT&E  evaluation 


10 
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Title  10  Allows  Support  to  OT&E 


DAG  Chapter  9 

Integrating  T&E  consists  of  many  aspects,  all  designed  to  optimize  test  scope  and  minimize 
cost.  For  example,  separate  contractor  developmental  testing  might  be  combined  with 
governmental  developmental  test  and  evaluation,  with  control  being  exercised  by  a  combined 
test  organization.  Live  testing  might  be  integrated  with  verified,  validated,  and  accredited 

Army  DA  PAM  73-1  *  30  May  2003 

c.  Discussions  with  system  contractor  personnel  may  be  necessary  to  ensure  full  technical  understanding  of  test 
incidents  observed  during  the  10 T&E  or  related  activities.  All  discussions  will  be  held  separately  from  any  scoring  or 
assessment  activities.  The  MATDEV  should  maintain  written  record  of  the  nature  of  these  contractor  and  Government 
discussions. 

Navv  OT&E  Framework  Intearated  Test  Methodoloav 

1l1iiL  i"  t  c_'  c.  i  blends  or-  c  ombine  s  contractor,  de  v  e  Lopmental.  and  O'  JL  Lo  form  -lv  cohesive 

testing  contimmm.  1  lvi s  integration  mm t  occui-  unless  participants  t  CT  I  ,  i  i -irl  C  >  1  i  laua-ve 

determined  (  1 1  e  i  t  entering  requirements  for  r-tdei:,]  u.es.1  testing  <.»f~  1  lio  system  under  evaluation.  IT  do  os 
n.ot  remo  ve  or  combine  *y  of  O I3-  _L  JE.’V/'  Jr  0 ’ s  cuixent  or  future  neq^uirements  for  reporting  Hn-jas-o-d  oil 
m  '-.eparare  <  C  > T*  J  T  ! analysis  of  the  sliared  test:  information  produced  l^y^  tire  ITT  effort. 
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5,6,2.  System  Contractor  Support  to  Operational  Testing.  System  contractors  may  be  beneficial  in 
providing  logistic  support  and  training,  test  failure  analyses,  test  data,  and  unique  software  and  instru¬ 
mentation  support  that  could  increase  the  value  of  operational  test  data.  Explanations  of  how  this  eon- 


Title  10: 

Contractor  cannot  be  involved  in: 

•OT&E  conduct 
•Establishing  OT&E  criteria 
•OT&E  evaluation 


Contractor  CAN  provide: 

•  Technical  understanding  of  test  incidents 

•  Logistic  support  and  training 

•  Support  to  test  failure  analysis 

•  Unique  software  and  instrumentation  support 

11 
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Title  10  Allows  Sharing  of  Data 


Army  DA  PAM  73-1  -  3D  May  2003 

a.  The  T&E  WIPT  goals  are  to  develop  a  mutually  agreeable  T&E  program  that  will  provide  the  necessary  data  for 
evaluations.  T&E  WIPTs  provide  support  for  the  development,  staffing,  coordination,  and  approval  of  all  required  T&E 


(6)  Support  the  CE  process  by  accomplishing  early,  more  detailed,  and  continuing  T&E  documentation,  planning, 
integration,  and  promote  the  sharing  of  data. 

T&E  Management  Guide 


4  8  CONTRACTOR  TESTING 

The  Deputy  for  T&E  is  responsible  for  ensuring 
that  contractor- conducted  tests  are  monitored  by 
the  government.  The  Deputy'  for  T&E  must  also 
be  given  access  to  all  contractor  internal  data, 
test  results,  and  test  reports  related  to  the  acqui¬ 
sition  program.  Usually,  the  contract  requires  that 
government  representatives  be  informed  ahead  of 
time  of  any  (significant  or  otherwise)  testing  the 
contractor  conducts  so  the  government  can 
arrange  to  witness  certain  testing  or  receive  results 
of  the  tests.  Further,  the  contractor’s  internal  data 
should  be  available  as  a  contract  provision.  The 
Deputy  for  T&E  must  ensure  that  government  test 
personnel  (DT&E  OT&E)  have  access  to  contrac¬ 
tor  test  results.  It  would  be  desirable  to  have  all 
testers  observe  some  contractor  tests  to  help  de¬ 
velop  confidence  in  the  results  and  identify  ar¬ 
eas  of  risk. 


Navy  OT&E  Framework  Integrated  Test  Methodology 

*  Fourth.  OT  uses  the  shared  data  from  the  IT  period  to  “answer”  or  achieve  resolution  on  as 
many  measures  of  effectiveness  (MOE)  and  measures  of  suitability  (MOS)  as  possible.  The 
goal  being  to  have  sufficient  data/test  information  at  the  end  of  the  IT  phase  to  resolve  most 
CGIs,  pending  successful  completion  of  the  final  independent  OT  phase. 

The  product  of  the  IT  integration  effort  should  be  an  IT  database,  similar  in  structure  and  content  to 
the  OT&E  Framework  database  (step  nine),  but  merged  with  DT  and  CT  requirements. 

Air  Force  AFI99-1D3  26  FEBRUARY  2008 

Operational  testers  may  use  data  from  sources  such  as  DT&E.  integrated  testing,  and  OAs  to  augment  or 
reduce  the  scope  of  dedicated  operational  testing  if  the  data  can  be  verified  as  accurate  and  applicable. 

5.5.2.  Contractor  T&E  Data.  Test  teams  and  TIPTs  should  use  as  much  contractor  T&E  data  as  possi¬ 
ble  if  its  accuracy  can  be  verified.  Contractor  T&E  data  should  be  visible  in  the  common  T&E  data¬ 
base. 

AFMAX63  119  20  JUNE  2008 

A 12. 1.2.  DT&E  and  OT&E  plans  and  concepts  are  structured  so  that  OT&E  can  capture  and  use 
DT&E  data  to  reduce  OT&E  requirements.  (ITT)  (See  A14,  A15,  A23,  A27) 


OSD  McQueary/Young  Memo  22  Dec  2007 

•  To  maximize  the  efficiency  of  the  T&E  process  and  more  effectively  integrate 

developmental  and  operational  T&E,  evaluations  shall  take  into  account  all  available 
and  relevant  data  and  information  from  contractor  and  government  sources. 
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T&E  Strategies 


5000.02 

b.  The  PM,  in  conceit  with  the  user  and  the  T&E  community,  shall  coordinate  DT&E. 

OT&E.  LFT&E,  family-of-sy stems  interoperability  testing,  information  assurance  testing,  and 
modeling  and  simulation  (M&S)  activities,  into  an  efficient  continuum,  closely  integrated  with 
requirements  definition  and  systems  design  and  development.  The  T&E  strategy  shall  provide 
information  about  risk  and  risk  mitigation,  provide  empirical  data  to  validate  models  and 
simulations,  evaluate  technical  performance  and  system  maturity,  and  determine  whether 
systems  are  operationally  effective,  suitable,  and  survivable  against  the  threat  detailed  in  the 
STAR  or  STA.  The  T&E  strategy  shall  also  address  development  and  assessment  of  the 
weapons  support  equipment  during  the  EMD  Phase,  and  into  production,  to  ensure  satisfactory 
test  system  measurement  performance,  calibration  traceability  and  support,  required  diagnostics, 
and  safety.  Adequate  time  and  resources  shall  be  planned  to  support  pre-test  predictions  and 
post-test  reconciliation  of  models  and  test  results,  for  all  major  test  events.  The  PM.  in  concert 
with  the  user  and  the  T&E  community,  shall  provide  safety  releases  (to  include  formal 
Environment,  Safety,  and  Occupational  Health  (ESOH)  risk  acceptance  in  accordance  with 
Section  6  of  Enclosure  12)  to  the  developmental  and  operational  testers  prior  to  any  test  using 
personnel.. 

Test  and  Evaluation  Strategy: 

•  Knowledge  to  manage  risks 

•  Empirical  data  to  validate  models  and  simulations 

•  Evaluate  technical  performance 

•  Evaluate  system  maturity 

•  Determine  operational 

•  Effectiveness 

•  Suitability 

•  Survivability 
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Implementation  Framework 

*  Integrate  the  People 

-  Integrated  Test  Teams 

-  Coordination  and  cooperation  for  integrated  strategy 

-  Early  OT&E  influence  on  test  design  and  scenarios 

*  Integrate  the  Planning 

-  Early  and  collaborative  planning  for  efficient  use  of  test  assets 

-  Improve  test  efficiency  and  streamline  test  schedule 

-  Reduce  duplication  and  voids 

*  Integrate  the  Data 

-  Maximize  data  available  and  usability  for  OT&E 

-  Common  data  formats  to  facilitate  sharing 

-  Incorporate  operational  realism  in  DT&E 
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Path  Forward 
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•  NDIA  System  Engineering  Division 

-  Tasked  DT&E  Committee  to  focus  on  Integrated  Testing 

-  “Starting  with  the  recommendation  from  the  2007-08  white  paper,  develop 
more  detail  on  methods  and  practices  for  Integrated  Test.” 

•  Integrated  CT/DT/OT  Committee  Approach 

-  Identify  existing  policies,  methods,  and  practices 

-  Determine  barriers  to  Integrated  CT/DT/OT 

•  Data,  resources,  planning 

•  Cultural  constraints 

-  Identify  potential  collaborative  approaches  within  current  policies 

•  Roles  and  positions  for  the  people  involved 

•  Interactions  between  stakeholders 

•  When  in  lifecycle  are  these  resources  involved 

•  What  are  the  output  products  of  each  interaction 
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Attributes  of  Integrated  Testing 


•  If  you  find  the  contractor  data  augmenting  the  OT&E  data, 

you  might  be  doing  integrated  testing 

•  If  the  DT&E  and  OT&E  personnel  recognize  each  other  in  the  airport, 

you  might  be  doing  integrated  testing 

•  If  the  OT&E  personnel  influences  DT&E  scenarios, 

you  might  be  doing  integrated  testing 

•  If  the  DT&E  system  is  operated  by  end  users, 

you  might  be  doing  integrated  testing 

•  If  the  CT,  DT,  and  OT  teams  are  sharing  data  in  a  common  format, 

you  might  be  doing  integrated  testing 

•  If  the  OT&E  confirms  DT&E  results, 

you  might  be  doing  integrated  testing 
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Summary 
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•  Integrated  Testing  is  Needed 

-  Facilitate  early  identification  and  correction  of  system  deficiencies 

-  Make  OT&E  a  process  of  confirmation  instead  of  discovery 

-  Minimize  “surprises”  when  the  product  is  sent  to  the  war  fighter 

-  Reduce  cost  and  schedule  with  shared  resources  and  reduced  duplication 

•  Title  10 

-  Prohibits  contractor  involvement  in  OT&E  conduct,  criteria  establishment,  or 
evaluation 

-  Allows  contractor  to  provide  technical  understanding  and  support 

-  Allows  for  collaborative  planning  and  execution  of  an  integrated  test 
program  to  provide  shared  data  to  support  independent  analysis 

•  Integrated  Test  Implementation  Framework  Involves  Integrating 

-  People:  Integrated  test  teams  to  introduce  operational  realism  earlier 

-  Planning:  Early  and  collaborative  efforts  to  streamline  test  program 

-  Data:  Sharing  of  data  to  address  developmental  and  operational  issues 
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